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^Thie  technical  report  eummarizes  the  image  understanding  and  image  processing 
research  activities  performed  by  the  Image  Processing  Institute  at  the  University  j 
of  Soitj^ern  California, during  the  period  of  1  April  1976  to  September  30,  1976 
under  contract  numbe”r  F-33615-76-C- 1203  with  the  Advanced  Research  Projects  i 
Agency  Information  Processing  Techniques  Office.  j 

"'The  research' program  has  as  its  primary  purpose  the  development  of  technique, 
and  systems  for  processing,  transmitting,  and  analyzing  images  and  two  dimension 
data  arrays.  Four  tasks  are  reported:  Image  Understanding  Projects,  Image  Pro-| 
cessing  Projects,  Smart  Sensor  Projects;  Recent  Ph.  D.  Dissertations.  The  Image  | 
Understanding  project  has  concentrated  on  edge  detectors,  ima^«  segmXntors,  | 

clustering  techniques  and  higher  level  symbolic  descriptors  from  segm^ted  image! 
The  Image  Processing  projects  have  concentrated  on  a  posteriori  restoration,  degl 
of  freedom  analyses  of  imaging  systems,  phase  coding  for  optical  image  ^rocessin 
and  artificial  stereo  algorithms.  The  Smart  Sensor  projects  include  the  ilevelop-  | 
ment  of  adaptive  CCD  image  processing  chips  and  circuitry.  The  Ph.  D.  disSertatiol 
completed  in  this  time  period  are  abstracted  in  the  next  task. 
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Scene  Analysis,  Image  Understanding,  Edge  Detection,  Image  Segmentation,  CoJD  ; 
Arrays,  CCD  Processors.  j 
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ABSTRACT 
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This  technical  report  summarizes  the  image  understanding  and 
image  processing  research  activities  performed  by  the  Image  Processing  j 

Institute  at  the  University  of  Southern  California  during  the  period  i 

of  1  April  1976  to  September  30,  1976  under  contract  number 
F-33615-76-C- 1203  with  the  Advanced  Research  Projects  Agency 
Information  Processing  Techniques  Office. 

The  research  program  has  as  its  primary  purpose  the  development 
of  techniques  and  systems  for  processing,  transmitting,  and  analyzing 
images  and  two  dimensional  data  arrays.  Four  tasks  are  reported: 

Image  Understanding  Projects;  Image  Processing  Projects;  Smart  Sensor 
Projects;  Recent  Ph.  D.  Dissertations.  The  Image  Understanding  project 
has  concentrated  on  edge  detectors,  image  segmentors,  clustering 
techniques  and  higher  level  symbolic  descriptors  from  segmented  imagery. 

The  Image  Processing  projects  have  concentrated  on  a  posteriori  restora¬ 
tion,  degree  of  freedom  analyses  of  imaging  systems,  phase  coding  for 
optical  image  processing  and  artificial  stereo  algorithms.  The  Smart 
Sensor  projects  include  the  development  of  adaptive  CCD  image  processing 
chips  and  circuitry.  The  Ph.  D.  dissertations  completed  in  this  time 
period  are  abstracted  in  the  next  task. 
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RESFAr?CH  PROJECT  OVERVIEW 


1. 


This  report  describes  the  progress  and  results  of  the 
of  Southern  California  image  understanding  and  image 
research  study  for  the  period  1  April  1976  to  30  September 
research  study  has  been  subdivided  into  five  projects: 

Image  Onderstanding  Projects 
Image  Processing  Projects 
Smart  Sensors  Projects 
Recent  Ph.D.  Dissertations 
Pub lications 


Universit 
processin 
1976.  Th 


The  image  understanding  projects  involve  research  directed  tovard  the 
goal  of  developing  generalized  processing  systems  capable  of  analyzing 
images  and  extracting  salient  information.  Specific  studies  include 
feature  extraction,  symbolic  description,  interpretation,  clustering 
techniques,  edge  detectors,  segmentors,  and  systems  analysis.  The 
image  processing  projects  include  research  on  novel  image  coding 
techniques  based  upon  results  of  the  image  understanding  study, 
advanced  image  restoration  methods,  including  a  posteriori  phase 
computation  techniques,  degree  of  freedom,  modelling  in  support  of 

I 

image  understanding,  and  studies  of  phase  coding  for  optical  filtering 
techniques  and  artificial  stereo  algorithms.  The  smart  sensors 
projects  comprise  investigations  of  electronic  and  optical  processing 
methods  which  can  be  integrated  with  imaging  sensors  to  perform  low 
level  image  enhancement  and  feature  extraction  within  the  sensor.  An 
adaptive  CCD  processor  has  been  developed  for  first  stage  image 


/ 


procassinj.  The  smart  sansors  research  warlt  is  being  performed  by  the 
Hughes  Aircraft  Company  Research  Laboratories  in  Malibu,  California 
under  subcontract  to  U5C. 

Section  2  of  this  report  details  the  research  effort  on  the  image 
understanding  projects.  The  image  processing  research  activities  are 
described  in  Section  3,  and  Section  4  covers  the  work  on  the  smart 
sensors  projects.  Section  5  is  a  compendium  of  abstracts  of  recent 
Ph.D.  lisserta tions  covered  under  this  past  six  month  period  and 
Section  6  is  a  list  of  publications  by  project  members  during  the 
reporting  period. 
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2.  IMAGE  OWDERSTANDING  PROJECTS 

The  results  of  the  ieage  understanding  projects  described  beloM 
are  directed  toward  the  eventual  developaent  of  processing  systeas  for 
generalized  iaage  analysis.  Applications  of  such  systeas  include 
photointerpretation,  vehicle  guidance,  visual  tracking,  cartography, 
and  nany  other  areas  of  ailitary  interest. 

Our  research  study  is  organized  according  to  the  structure  of  a 
conceptual  iaage  understanding  systea.  This  systea  consists  of  a  . 
feature  extraction  stage  which  detects  and  aeasures  priaitive  features  j 

5 

such  as  edges  and  texture  regions  froa  an  input  iaage.  Next  the  ^ 
priaitive  features  ara  clustered  into  aeaningful  syabols  such  as  ! 
object  boundaries,  s^egaents  of  soae  coaaon  attribute,  or  basic  shape 
structures.  Such  clustering  then  is  used  to  coapletely  segaent  the 
scene  of  interest.  Finally  iaage  syabols  are  interpreted  in  teras  of  < 
their  seaantic  relationships  to  produce  a  concise  guantitative 
description  of  the  original  iaage.  Soae  extra-iaage  knowledge  base  is 
assuaed  available  for  guidance  of  all  eleaents  of  the  iaage 
understanding  systea. 

The  philosophic  approach  to  the  study  has  been  to  bring  together  | 
a  research  teaa  skilled  in  digital  signal  processing  and  concepts  of 
artificial  intelligence.  Although  the  research  effort  at  USC  involves 
all  stages  of  the  overall  iaage  understanding  systea,  eaphasis  has 
been  placed  on  the  feature  extraction,  clustering,  and  syabolic 
description  subsystea  eleaents.  Rapid  progress  is  being  aade  in  these 
areas  to  assist  the  research  in  the  higher  stages  of  the  iaage 
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2.1  BWLOATrON  OF  A  SIBPLIFIED  HUECKEL  OPEBATOB 


Raaakant  Nevatia 


Intcoduction 


An  edge  detector  due  to  Hueckel  <1,2>  has  been  in  wide  use. 
though  no  definitive  studies  have  been  nade,  its  pecforaance  is 
considered  to  be  superior  to  that  of  other  edge  detectors  (e.g.  see 
<3>,  see  <4>  for  a  contrary  conclusion) .  However,  conputation 
required  for  a  Hueckel  edge  detector  is  considerably  larger  than  for 
other  types  of  operators.  The  central  part  of  the  Hueckel  edge 
detector  involves  approxiaation  of  a  two  diaensional  inage  intensity 
pattern  by  expansion  into  a  truncated  Fourier  series.  Hero  and  Vassy 
described  a  nethod  of  siaplifying  the  approxiaation  by  using  fewer 
bases  functions  for  expansion  <5>.  In  this  report,  we  exaaine  a 
generalization  of  this  approach  and  effect  cn  perfornance  of  the 
reduction  in  the  nunber  of  bases  functions.  The  aain  experinental 
conclusion  is  that  such  reduction  in  conputation  is  achieved  at 
significant  loss  in  perfornance,  if  the  inages  in  use  are  noisy. 


Operation  of  the  Hueckel  Edge -Line  Detector 


The  basic  process  of  Hueckel  edge  operator  consists  of  optinally 
fitting  an  ideal  edge-line  to  the  inage  intensity  values  in  a  snail 
circular  neighborhood.  The  ideal  edge-line  is  deternined  by  a  7-tuple 
of  paraneters,  three  paraneters  deternining  the  brightness  levels  (b 
t  and  t^,  as  shown  in  figure  1)  and  the  other  four  paraneters 
deternine  the  position,  orientation  and  the  width  of  the  line.  Tl 


fitting  process  consists  of  detersining  the  values  of  the  seven 
paraseters  for  a  best  fit  vith  the  iaage  intensities.  Ideally  a  tuple 
of  paraaeters  is  to  be  coaputed  such  that 

N  =  II  I  -  S(tuple)  11  (1) 

is  a  ffliniaua,  where  I  is  a  vector  representing  iaage  intensities  and  S 
is  an  ideal  edge-line. 

This  miniaization  process  is  approziaated  ty  expansion  of  both 
the  input  iaage  disk  and  the  edge-line  in  an  orthogonal  Fourier 
series.  Let  a.  be  the  coefficients  of  expansion  for  the  iaage  and  s.  |j 
be  the  coefficients  for  an  ideal  edge-line.  The  ainiaization  of  (1)  !| 
is  then  approxiaated  by  choosing  a  tuple  such  that 

8 

i=0 

is  ainiaized.  Hueckel  gives  arguaents  for  the  optiaality  of  the 
chosen  series  <2>.  A  crucial  decision  is  to  use  only  the  first  nine  ji 
teras  of  expansion.  Two  reasons  are  given  for  the  choice  of  this 
nuaber: 

(a)  Higher  order  teras  correspond  to  noise  in  the  iaage  and 

should  be  ignored.  j 

(b)  An  analytical  solution  to  the  ainiaization  problea  is  found  ' 
using  nine  teras. 

Decision  as  to  the  presence  of  an  edge  is  based  on  the  aaplitude 


of  the  confuted  step,  as  well  as  the  degree  of  fit,  deterained  bj  N  in 
eq.  (1)  above. 

S iaplif ica tion  of  the  Operator 

The  coaputation  of  Hueckel  operator  can  be  reduced  substantially 
if  the  nuaber  of  teras  used  in  the  expansions  above  is  reduced.  An 
edge-line  is  completely  specified  by  seven  paraaeters.  Further, 
absolute  brightness  level  (b  )  is  of  little  interest  for  edge 
coaputation;  only  six  Fourier  coefficients  are  needed  to  coapute  these 
paraaeters.  Also,  the  problea  of  alniaization  in  eg.  (2)  above  reduces 
to  that  of  determining  six  unknowns  in  six  equations  (details  are  in 
the  Appendix)  . 

Only  four  parameters  (orientation,  position  and  step  size)  are 
needed  if  only  step  edges  are  of  interest.  Hero  and  Vassey  describe 
an  approach  to  determining  these  paraaeters  for  a  square  region  of  the 
image  <5>.  They  conclude  that  expansion  in  only  two  bases  functions 
is  needed  to  deteraine  the  orientation  of  the  step.  The  position  and 
step  size  are  then  deterained  by  an  approximation. 

We  have  generalized  the  procedure  to  detecaine  the  four 
paraaeters  for  a  step  from  four  coefficients  of  expansion,  used  by 
Hueckel,  for  a  circular  neighborhood  and  deteraioing  the  six 
paraaeters  for  a  step-lina  from  six  coefficients  of  expansion. 

In  either  case,  as  far  as  the  truncated  expansion  is  concerned, 
the  signal  can  be  fit  perfectly  with  an  ideal  edge  or  an  ideal 
edge-line.  Thus,  no  ainimization  coaputation  is  necessary. 


The  iapoctant  question  is  the  effect  of  ignoring  soae  tens  in 
the  orthogonal  expansion  of  the  signal.  It  is  clear,  that  if  the 
signal  indeed  consists  of  an  ideal  edge  eleeeot,  the  fewer  teres  of 
expansion  are  sufficient  to  characterize  the  signal  exactly.  Results 
of  experinental  evaluation  signals  containing  noise  are  presented 
next. 

gxperieental  Results 


The  input  for  these  experieents  consists  of  a  picture  with  two 
vertical  edges;  the  intensity  profile  is  as  illustrated  in  figure  2. 
A  varying  aaount  of  randon,  Saussian  noise  is  added  to  this  iaage. 
Results  are  presented  for  three  ieages,  figures  3a,  4a  and  5a,  with 
step  size  to  noise  variance  ratios  of:  10,  4  and  2. 

The  three  inages  were  processed  by  three  versions  of  the  edge 
detector; 


1.  Original  Hueckel  edge  detector  (operator  1) 

2.  Edge  line  detector  using  six  coefficients  (operator  2) 

3.  Edge  detector  using  four  coefficients  (operator  3) 

Deteriining  the  presence  of  an  edge  requires  the  use  of  a 
threshold  on  the  computed  step  amplitude.  Use  of  a  higher  threshold 
will  allow  fewer  noise  edges  but  also  lose  seme  of  the  desired  edges. 
For  the  results  presented,  the  thresholds  were  determined  for  the  best 
subjective  performance.  The  differences  in  the  performance  of  three 


operators 


are  so  marked  that  a  more  careful  control  of  thresholds 


Figure  4.  Medium  noise  image  and  detected  edges 


Figure  Hirh  noise  image  and  detected  edges 


(e.g.  by  requiring  the  total  nunber  of  detected  edges  to  be  the  same) 
was  not  considered  necessary. 

Figures  3,4  and  5  show  the  results  of  the  three  operators  on 
three  ioages.  (The  edge  detector  was  applied  every  three  pixels  along 
every  third  row.  Each  application  looks  at  a  neighborhood 
approxinately  circular  and  eight  pixels  in  diaaeter.  There  ix 
considerable  overlap  of  regions  in  two  neighboring  applications, 
resulting  in  nultiple  edges  for  the  saae  step.  The  phenoaenon  is 
pronounced  in  figures  3,  4  and  5  for  the  right  edge  only,  because  of 
accidental  placements.) 

It  is  clear  that  the  perforaance  of  operator  1  declines  gradually 
with  the  increase  in  added  noise.  Operators  2  and  3  on  the  other 
hand,  perfora  reasonably  well  for  high  signal  to  noise  ratio,  but  very 
poorly  for  images  with  high  noise.  Further,  the  perforaance  of 
operator  2,  using  more  coefficients  of  expansion,  is  superior  to  that 
of  operator  3.  (In  absence  of  any  noise,  all  three  operators  perfori 
perf  ec  tly. ) 

Siailar  results  are  obtained  for  detection  of  line  edges  (i.e. 
when  the  width  of  step  is  small,  say  two  or  three  pixels  wide) . 

Conclusions 

The  results  shown  above  are  soaewhat  surprising.  It  is  tempting 
to  think  that  as  the  ideal  step  is  coapletely  deterained  by  foul 
paraaeters,  four  bases  vectors  can  be  found  to  span  the  space  of  idea! 
step  elements.  Unfortunately,  the  space  spanned  by  the  ideal  steps  il 
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not  a  complete  space  (the  addition  of  tuo  arbitrary  steps  is  not 


another  step).  (For  the  special  case  of  edges  defined  on  a  3  x  3 
square  neighborhood,  a  complete  two  dimensional  sub-space  spanned  by 
the  edges  has  been  determined  by  Prei  and  Chen  <6>) . 


It  is  concluded  that  the  simplification  cf  Hueckel  edge  detector, 
by  ignoring  some  of  the  terms  of  Fourier  expansion  results  in 

unacceptable  loss  in  performance,  if  images  are  likely  to  be  noisy. 
(Unfortunately,  Hero  and  Vassey  present  no  results  cn  comparable 
images  in  their  claim  that  the  performance  is  not  coiprcmised. ) 

Further,  it  raises  questions  as  to  the  effect  of  using  only  nine  terms 
in  Hueckel's  original  operator  and  also  the  effects  of  similar 

approximations  using  other  series  for  expansions,  (e.g.  see  <7>)  , 

Appendix;  Details  of  approximations 

Familiarity  with  references  <1,2>  is  assumed  here  and  the 
notation  used  therein  is  used  here  without  much  elaboration.  The  nine 
coefficients  of  expansion  for  an  ideal  edge-element  are  as  follows: 

Sq  =  (3TT)^(3Zb  +  t  (16  -  21r_  +  7r^  -  7r^  +  5r^) 

,  c  7  (3) 

+  (16  -  21r^  +  7t^  -  7r^  +  5r^)  /  56 

s,  =  X  r  +  X  r 

(«) 


*2  = 


(5) 


“3  =  ^  + 


(6) 
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(7) 


.4  =  2MX_r_  +  X^r_^)(c^.cJ) 


S5  =  8^(X_r_  +  X^r^)c^c^ 
"6  =  + 


s_  =  5^ (X  r^  +  X . rf ) c 
7  -  -  +  +  y 


s„  =  X  r  (0.5-2.  5r^)  +  X^r  (0.  5  -  2.  5r^) 


where  r  ,  r^  stand  for  the  positions  of  brightness  transitions,  b  , 

4  2  2 

and  t  are  as  shown  in  figure  1,  X  =  (3n)‘^  t.  (1-r.  )/4,  and  c  and 

+  i  i  i  * 

are  the  direction  cosines  of  the  edge  orientation  (see  <1>  f 

details) .  Consider  two  cases  separately; 


1.  Step  edges  only: 


The  ideal  edge-line  can  be  converted  to  a  step  by  setti 

t  =0  (hence  X  =0).  He  are  now  interested  in  deteraining  X 
+  '  +  • 

r  ,  c  and  c  .  8e  need  use  only  egs. (4) ,  (5)  and  (6)  and  hen 
-  X  y 

need  only  three  coefficients. 


2.  Step  and  line  edges: 


Six  paraaeters  need  to  be  deterained  now  and  we  need  s 

equations.  Several  subsets  of  the  above  nine  equations  wi 

suffice.  In  particular,  egs.  (4)  through  (9)  are  sufficient.  T 

solution  can  proceed  by  deteraining  c  and  c  froi  eqs.  (5)  a 

X  y 
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(6)  (note:  *  Paraaeters  ^  »  r_  and  can  now 

be  determined  from  four  equations:  (4),  (5)  or  (6),  (7)  or  (fl)  , 

and  (7)  or  (10)  .  These  equations  can  be  solved  in  exactly  the 

same  way  as  in  reference  <1>,  Appendix  A.  (Note  that  computation 

of  c  and  c  using  all  nine  coefficients  requires  iterative 
X  y 

solution  of  a  fourth  order  po  ly  no  ■ia 1. ) 
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2.2  APPENDIMG  OF  LINEAR  EDGE  SEGMENTS  MHICH  CCMTAIM  CCHHCN  EDGE  POINTS 
Lee  U.  Martin 

In  an  iaage,  there  are  generally  sub-areas  which  stand  out  froe 
the  background.  They  aay  be  classifiable  as  straight  lines,  curves, 
boundaries,  etc.  and  they  aay  be  given  the  attributes  of  a  feature, 
i.e.  soaething  that  can  be  used  in  describing  the  iaage  scene,  using 
a  set  of  iaage  pixels  as  a  single  eleaent.  That  eleaent,  or  feature, 
has  a  wore  global  inforaation  content  than  the  stea  of  the  pixels 
because  of  a  recognizable  interpixel  relationship.  This  report 
considers  the  appending  of  linear  segaents  to  create  longer,  and 
possibly  aore  identifiable  edge  segaents  to  te  used  as  features. 

Elongated  edge  segaents  generally  correspond  to  areas  or  objects 
of  interest  in  a  scene,  e. g.  partial  loundaries  of  objects, 
identifiable  segaents  along  an  edge  (perhaps  functions,  or  particular 
occurrence  of  an  edge  pattern).  As  such  this  feature  would  be  useful 
for  stereo  analysis  (saall  angle  stereo)  or  scene  segaentation.  In 
the  latter  deternine  areas  of  the  iaage  where  further  (second  level) 
processing  would  be  aost  fruitful,  e.g.  deteraine  segaent 
cont inuabllity  criteria  deterained  by  the  feature  segaent.  In  saall 
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angle  stereo  analysis,  features  in  a  priaary  iaage  will  generally  have 
a  high  correspondence  with  a  segaent  edge  which  lies  on  siailac  points 
in  the  coapleaent  iaage,  allowing  identification  of  correspondent  edge 
segaents.  These  elongated  edge  segaents  satisfy  soae  criteria  for 
being  selected  as  features  as  previously  stated  <1>.  They  have 

(a)  relative  invariance  under  slightly  dissiailar  iaaging  conditions 
and  saall  perspective  changes. 

(b)  reproducibility  of  the  feature 

(c)  uniqueness,  or  at  least  the  set  of  features  which  have  coaaon 
attributes  should  be  saall,  to  be  useful  in  ccaplenent  iaage  Batching 

(d)  correspondence  with  areas  of  interest  in  the  iaage,  e.g.  object 
boundaries. 

Reference  <1>  describes  the  steps  leading  to  the  attainaent  of  a 
linear  segment  nap.  Basically  a  Huechel  <2>  edge  operator  processes 
the  entire  iaage  returning  an  edge  point  aap,  which  in  turn  is 
processed  to  link  edges  within  orientation  and  proxiaity  constraints 
<3>.  At  this  point,  edge  segments  which  are  continuous  in  the  iaage 
are  broken  into  subsegaents  which  lie  along  pre-selected  orientations. 
Generally,  if  the  edge  curve  is  saooth,  edge  pcints  near  the  end  of 
linear  'segaents  will  also  occur  in  linear  segments  occurring  in  the 
sane,  or  adjacent  angles.  Thus  we  have  "points  of  continuity"  with 
which  we  may  link  linear  edge  segaents  to  create  more  general  edge 
segments  of  greater  length. 

The  typical  links  possible  are  shown  in  figure  1.  Segaents  are 
classified  as 


-19- 


(lb)  Typical  connection,  when  H  and  T 
are  in  different  PS's. 


(Ic)  Typical  case  where  H,  T  are  in  same  PS.  Then 
CS  must  be  absorbed  into  SI. 


Figure  1.  Append  Schematics 


(1)  previous  segnents  (PS) 

(2)  candidate  segments  (CS)  . 

The  ZS  may  be  appeniel  to  any  PS  with  a  common  edge  point.  Only  the 
head  (H)  and/or  the  tail  (T)  of  the  CS  may  be  in  common  with  some  PS. 
Each  edge  point  in  the  set  of  edge  points  in  the  PS's  (previously  used 
edges,  PUE)  belongs  to  a  unique  PS,  but  may  re-occur  in  some  future 
CS.  A  PS  contains 

(1)  the  H  of  CS  designated  as  Si, 

(2)  the  T  of  CS  as  S2. 

Notation  HS2  implies  the  head  of  segment  2,  i.e.  the  PS  which 
contains  T(of  CS).  D,  (R) ,  preceding  a  segment  identifier  implies 
possible  disconnect  subsegments,  (possible  remainder  subsegment  i.e. 
appended  to  CS) .  CS  is  appended  to  Si  (S2)  if  the  length  (CS)  > 
length  (DSI)  ,  I  =  1,2.  Otherwise  CS  is  identified  as  a  new  segment 
(given  a  new  segment  identifier  number) .  All  possible  linkages  were 
considered,  as  indicated  in  figure  1.  The  program  to  accomplish  the 
inter-angle  and  intra-angle  links  is  complete  and  approximately  95* 
debugged. 

The  preliminary  results  indicate  a  slight  angular  preference  in 
the  direction  of  the  step  angle  (♦cc,-c)  if  linking  is  not  done  over 
the  full  360  range  (because  links  can  only  be  made  to  PS's  which 
occur  at  previous  step  angles).  All  anticipated  modes  of  linking  have 
been  accomplished  with  the  desired  results. 

Upon  completion  of  the  programming  effort  the  results  will  be 
applied  to  stereo  pairs  for  evaluation. 
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2.3  CONTRIBUTION  OF  EDGE  ORIENTATION  OF  EDGE  DETECTION 

Peter  Chuan  j 

The  purpose  of  this  investigation  is  to  find  the  class  of  edge 

|. 

detectors  that  could  find  edges  with  greater  resolution.  One  of  the  ; 
properties  that  could  be  looked  into  is  the  orientation  of  edges  in 
the  inage.  The  pictures  we  are  aost  concerned  with  are  pictures  of 
natural  iaages,  aost  of  which  contain  structured  objects  like  trucks 
and  vehicles  on  a  natural  background.  Figure  1  shows  the  picture  of  a 
tank  with  a  background  of  aarsh.  This  background,  for  our  purposes  of 
extracting  only  the  tank,  is  considered  noise.  However,  it  should  be 
noticed  that  this  very  background  which  we  condean  to  be  noise 

i 

actually  contains  considerable  inforaation  in  itself.  Therefore,  any 


good  edge  operator  that  will  pick  up  high  resclution  edges  will  detect 
both  background  and  object.  The  problea  facing  all  edge  detectors  is 
to  be  able  to  selectively  discriainate  edge  points  not  of  interest. 
For  this,  locally  adaptive  thresholding  is  considered. 

An  edge  aagnitude  aap  and  an  edge  orientation  lap  is  produced. 

Figure  2  shows  the  edge  nap  thresholded  so  that  the  aost  significant 

4000  edge  points  are  displayed.  In  this  ezaaple,  the  Rirsch  operator 

is  used.  Based  on  the  observation  that  edge  points  which  do  not 

belong  to  any  structured  object  should  have  edge  directions 

2 

distributed  randonly,  the  variance  o  of  edge  directions  inside  a 
3  1  3  window  is  conputed.  The  edge  aagnitude  nap  e(x,y)  is  then 
aodified  by  the  following  relation 

and  the  resulting  aodified  edge  nap  e*{*»y)  is  thresholded  to  obtain 
figure  3. 

Other  infornation  that  could  be  obtained  froa  the  edge  direction 
nap  is  the  continuity  of  edge  directions.  In  figure  4,  the  edge  nap 
e*  (x,y)  is  thresholded,  where  e*  (x^y)  is  the  aodified  version  of  the 
edge  aagnitude  nap  e(x,y)  given  by  the  relation 

If  the  edge  orientation  at  point  (x,  y)  is  0  ( x,  y)  ,  and  the  orientation 
of  the  pixels  at  the  front  end  and  back  end  of  C(x,  y)  is 
0(x2»y2) »  respectively,  then 
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Figure  1.  Tank  on  a  noisy 
Background 


Figure  2,  Kirsch  thresholded 
to  display  4000  most 
significant 


i 


Figures  5a,  5b,  and  5c  show  the  same  three  procedures  applied  to 
a  noisy  straight  line  edge  an  1  evaluated  with  Pratt's  figure  of  aerit 
<1>.  Tlip  original  picture  of  the  line  has  an  S/N  ratio  of  five 
defined  in  the  convention  given  by  the  above  figure  of  merit. 

The  original  Kirsch  output  has  a  figure  of  merit  F  =  68*.  The 
Kirsch  output  inplemented  with  directional  variance  measure  has 
F  =  and  the  Kirsch  output  inplenented  with  continuity  modifying 

function  has  F  =  7*^%. 

From  figures  2,  3,  and  U  it  is  obvious  that  the  continuity  and 
variance  modifying  functions  create  more  background  edges  than  the 
original  edge  magnitude  map.  However,  figure  U  shows  that  more  edges 
on  the  side  of  the  tank  were  detected  by  the  continuity  modifying 
function  while  the  other  edges  are  more  or  less  preserved.  Figure  3 
shows  edges  that  have  less  random  edge  directions  and  therefore  most 
detected  edges  appear  as  part  of  a  chain  of  edge  points.  It  is  not 
exactly  clear  as  to  what  extent  the  orientation  information  could 
contribute  to  detecting  fine  edges. 
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Figure  5a.  Kirsch  output 
F  =  68% 
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Figure  5b.  Kirsch  with  variance 
modifying  function 
F  =  0.  51% 


Figure  5c.  Kirsch  with  continuity 
modifying  function 
F  =7  5% 
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2.4  SC’=‘N!=;  SEGMENTATION  BY  CLUSTERING 

Guy  Coleman _ _ _ _ _ 

The  major  portion  of  the  effort  during  the  period  of  this  report 
has  been  directed  at  examination  of  scene  features  and  their 
relationship  to  the  overall  clustering  problem.  The  only  intrinsic 
features  of  a  pixel  in  a  monochrome  image  are  its  position  (x  and  y 
coordinates)  and  its  brightness.  All  other  features  are  of  higher 
order,  that  is,  they  are  based  upon  the  relaticnship  of  the  pixel  to 
other  pixels  in  the  scene. 

A  number  of  different  features  were  computed  for  overlapping 
blocks  in  an  aerial  photograph  (see  figure  1).  As  expected,  the 
larger  block  sizes  virtually  destroy  the  scene  detail.  These  features 
(see  figures  2  through  6  for  examples)  were  computed  by  averaging  the 
value  of  the  feature  over  the  block. 

A  different  approach  was  adopted  for  the  tank  scene  (figure  7) 
and  the  armored  personnel  carrier  scene  (figure  8).  In  these  scenes, 
features  were  computed  for  several  block  sizes  centered  on  each  pixel. 
While  this  approach  requires  more  computation,  the  intrinsic 
resolution  of  the  picture  is  maintained  (see  figures  9  through  22  for 
examples) .  Some  of  the  features  (mean,  for  example)  still  cause 


substantial  blurring  and  loss  of  detail. 


Figure  1.  Aerial  photograph 
original 


Figure  4.  Texture  8x8 


Mode  8x8 


Median  8x8 


1,  •  y  ^  . 

Figure  7.  Tank  original 


Figure  9.  Tank  mean  7x7 


Tank  variance 


Figure  11.  Tank  texture  7x7 


Tank  mode  7x7 


Figure  17.  APC  Variance  7x7  Figure  18.  APC  Texture  7x7 


Soae  of  the  features  are  highly  influenced  by  the  value  of  a  i< 
pixels  within  the  block.  Exaaples  of  this  ace  variance  and  teztu 
(averaged  aodified  Roberts  Cross  operator).  These  features  alwa 
have  large  values  at  edge  boundaries.  Segnentations  based  on  the; 
features  would  therefore  be  expected  to  identify  "edge"  as  a  distin( 
region  in  the  segmented  inage. 

The  nedian  and  node  features  are  focned  by  coaputing  the  gn 
scale  histograa  nedian  and  node  in  a  block  centered  on  each  pixel  ; 
the  scene.  These  features  (node  in  particular)  do  not  cau; 
substantial  degradation  of  the  scene  resolution.  The  nedian  featui 
does  cause  soae  blurring  while  the  node  feature  fills  in  soae  conca' 
areas  and  clips  soae  highly  convex  areas.  Nevertheless,  feature 
subjected  to  nodal  filtering  are  expected  to  substantially  inprove  tl 
classifier  perfornaiice  since  they  are  based  on  the  nost  frequenti 
occurring  values  within  the  block  centered  cn  each  pixel.  Textui 
features  having  these  properties  are  being  investigated. 

The  activity  in  progress  at  present  is  concerned  with  featu 
thinning  as  a  preprocess  to  clustering.  The  feature  covariance  natr 
was  coaputed  and  subjected  to  Karbunen-Loeve  transfornation.  T 
resultant  features  are  linear  conbinations  cf  the  original  featur 
and  are  statistically  uncorrelated.  The  highly  correlated  set 
original  features  will  be  replaced  with  an  uncorrelated  set  that 
substantially  saaller.  Reduction  of  the  feature  space  dinension 
this  lanner  is  expected  to  reduce  the  anount  of  conputation  requir 
in  the  clustering  algorithe. 
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The  clustering  phase  of  the  scene  segaentation  is  presently  in  a 
prelininary  state.  It  is  anticipated  that  the  oferall  perforeance 
will  not  be  highly  dependent  on  the  clustering  algorithe  chosen  and 
the  algoritha  chosen  will  be  one  hawing  coaputational  efficiency.  The 
procedure  under  consideration  presently  will  proceed  as  follows  (see 
f  igure  23)  : 

1.  Initial  clustering  with  a  few  clusters  will  be  perforaed  using  the 
thinned  features  computed  previously. 

2.  The  "guality”  of  the  clustering  will  be  computed  by  any  of  several 
methods  presently  under  consideration. 

3.  The  riuaber  of  clusters  will  be  increased  (by  cne  perhaps)  and 
steps  1  a  nd  2  repeated. 

4.  The  process  will  end  when  the  iaproweaent  of  the  cluster  quality 
is  less  than  a  threshold. 

It  has  been  recognized  that  many  of  the  linguistic  or  artificial 
intelligence  methods  at  higher  levels  of  the  image  understanding 
systems  rely  on  preprocessing  which  produces  closed,  connected 
boundaries  of  the  scene  segments.  The  approach  described  here  is 
based  on  the  vector  space  model  and  uses  communication  theory  methods 
to  achieve  segmentation.  Because  of  that,  a  finite  ais-classif ication 
probability  will  always  exist.  It  is  likely  that  a  transitional 
procedure  will  be  required  to  produce  closed  connected  boundaries  from 
the  segmented  scene  before  further  processing  can  be  performed. 
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rocessmg  to  Categories 


2.5  RECURSIVE  RSTIMUTICN  OP  BOUNCRBIES 

Masser  E.  Mahi  and  Siion  Lopez  Hora _ 

The  purpose  of  this  research  is  to  develop  recursive  algorithas 
for  the  estiaatioD  of  boundaries  of  objects  in  noisy  pictures,  the 
available  inforaation  being  liaited  to  the  statistics  of  the  object 
boundary,  object  and  background. 

The  scanned  picture  in  the  absence  of  ncise  is  represented  by 
S(k)  =  X(k)S^(k)  +  (1-  Mk))Sj^(k)  (1) 

2 

for  k  =  1,...,M  where  M  is  the  nuaber  of  lines  in  the  picture,  s  and 

o 

s^  denote  the  intensity  values  of  object  and  background  assuaed  to  be 

saaple  functions  of  two  statistically  independent,  c yclo- stationary 

randoa  .  sequences  whose  first  two  aoaents  are  available,  and  X  is  a 

binary  valued  function  taking  values  of  1  or  0  corresponding  to  points 

of  the  iaage  belonging  to  the  object  or  the  background,  respectively. 
The  domains  of  S^,  Sj^  and  X  are  the  entire  picture. 

A  set  of  observations 

y(k)  =  S(k)  +  V(k)  (2) 

is  assuaed,  with  s(k)  as  defined  in  eq.(1)  and  v(k)  representing  a 

2 

zero  Bean  Gaussian  white  noise  sequence  of  variance  o  . 

Let.  us  assign  to  each  k  two  variables  w(k)  and  c  (k)  representing 
the  width  and  the  geometrical  center  (as  measured  frea  the  aidpoint  on 
the  iaage)  of  the  object  at  the  associated  line.  Hence 
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w(k+l)  =  w(k) 
c(k+l)  =  c(k) 


(m-l)N+l  s  k  s  m  N-1 

m  =  1,  ....  N 


Since  the  object  width  is  a  positive  quantity  and  the  object  lust 

be  within  the  iaage,  w  (k)  and  c(k)  satisfy 

w(k)  ^0 

w(k)/2  +  I  c(k)|  ^  N/2 

In  terms  of  the  widths  and  centers,  the  chosen  representation  for  ^ (k) 


X(k)  =  ^  (in-l)N+N/2  +  c(k)  -  w(k)/2  s  k  ^ 

lo  else  {m-l)N+N/2  +  c(k)  +  w(k)/2 


The  quantities  w(k)  and  c(k)  are  random  sequences  with  known  first  and 

second  order  statistics.  Let 

w(k)  =  w(k)  +  w  (6) 

c(k)  =  "cCk)  +  c 

where  w  and  c  are  the  mean  values,  respectively.  As  a  first  order 

approximation,  the  pair  r(k)  =  [w(k)  c(k)  ]•  is  assumed  to  represent  a 

first  order  Harkov  sequence, 

r(k+l)  =A  (k)  r(k)  +  B  (k)  u  (k)  (7) 

o  o  o 

where  u^tk)  =  [u^  ^2]'  is  a  zero  mean  white  normal  sequence  with  unit 
covariance,  A^(k)  and  B^(k)  are  matrices  cf  appropriate  dimensions. 


As  a  result  of  eq.(3) 


Ao(k)=  I 

B  (k)  =  0 
o 


(identity  matrix) 


for  all  k  except  k  =  sN,  «  =  1,...,N. 


By  augmenting  eq.(7)  with  dynamic  equations  representing  the 
statistics  of  the  object  and  background  it  is  possible  to  state  a 
nonlinear  NHS  estimation  problem  that  can  be  solved  by  the  wethod 
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i n t rod  us s d  in  <  1  > . 


To  illustrate  the  estimator  perforaance  a  binary  picture  with  an 
ellipse  of  unit  luminance  against  a  background  of  zero  luminance  has 
been  used.  The  original  observation  has  been  corrupted  by  adding 
white  r;aussian  noise  of  standard  deviation  0.5  (a  =  0.5  or  P  =  1.0)  in 
the  high  sig nal- to-noisa  ratio  case  and  noise  of  variance  4  (O  =  2.0 
or  p  =  0.25)  for  the  highly  noisy  case.  Two  different  estimators  were 
used:  I)  estimation  of  the  boundaries  when  s^  (k)  =  1  and  s^(k)  =0  in 

e<]*(1)»  this  condition  singles  out  the  boundary  estimation  itself,  see 
figures  1b,  1c,  2b,  and  2c.  II)  estimation  of  the  boundary  as  well  as 
the  object  intensity  when  the  last  one  is  assumed  constant  and  only  a 
priori  information  on  its  variance  and  mean  is  known,  see  figures  Id, 
1e,  2d,  and  2e.  In  either  case  a  refinement  of  the  estimator 

consisting  of  a  detection  test  at  the  end  cf  every  line  has  been 
performed.  As  a  result  of  this  detection  step,  the  estimate  can  be 
rejected  in  which  case  the  estimator  is  initialized  or  accepted 
without  change.  The  estimator  detector  form  is  referred  to  as  the 
refined  estimator. 

The  parameters  chosen  for  these  estimators  were: 

N  =  256  W  =  57  C  «  0 


Case  I 


A  (mN) 
o 


Case  II 


B  (mN) 
o 


-37- 


Best 

Available 

Copy 


( (.■  )  rcfinc<i  I'.stirnati'  (  >■  )  rofiin-ci  estimate 


1.  Kstiinatf'cl  anti  oriLjinal  ixmiHicaries  for  kimwii  (!),<•)  and 
unknown  (d.e)  otiiio  t  luioin.un  o.  ^  indii  ales  sicnal-to 
noise  ratio. 


(a)  the  observation,  p 


/.926  0  0  \  /7.78  0  0 

A(mN)=(  0  .97  0  1  B(mN)  =  (  0  2.91  0 

Ol'  \o  00 


E  s  =0.5 
o 


E(s  -  E  s  )  =  0.C5 

o  o 


Although  the  estimator  procedure  is  applicable  to  non-binary 
pictures,  the  way  to  reduce  the  possible  increase  in  computation  is 
still  under  investigation. 


Reference 
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2.6  SHAPE  ANALYSIS  AND  DESCRIPTION 


Erica  M.  Rounds 


This  research  is  directed  toward  constructing  region 
representations  which  are  useful  at  the  interpretation  level.  A  good 
representation  should  be  largely  invariant  with  respect  to  location, 
orientation,  and  size  to  facilitate  matching  with  object  descriptions 
in  the  knowledge  base  <3>.  The  process  of  extracting  global  features 
from  region  descriptions  is  called  here  "shape  analysis." 

Shape  analysis  can  be  considered  as  an  intermediate  level  between 
the  segmentor  and  the  interpreter.  Each  of  these  levels  deals  with 
image  descriptions  on  a  conceptually  different  basis.  Por  exanple. 
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regions  of  homogeneous  properties  extracteiJ  hy  the  segmcntor  are 
consirlered  areas  by  the  shape  analyzer  having  geometrical 
characteristics  such  as  size,  perimeter,  and  shape.  On  the 
interpretation  level  these  may  correspond  to  object  surfaces. 
Likewise,  global  discontinuities  in  the  contour  may  te  associated  with 
vertices  of  a  three-dimensional  object. 

The  shape  analyzer  operates  on  the  region  description  produced  by 
the  segmentation  level.  Typically,  this  consists  of  the  set  of  points 
which  constitutes  the  boundary  of  a  region.  More  precisely,  we  define 
the  closed  boundary  of  a  simple  region  (nc  holes)  to  be  a  set  of  N 
ordered  grid  points  such  that 

2  <  k  <  N,  and  =  (*i»yi)*  Geometrically,  the 

sequence  of  points  corresponds  to  a  clockwise  or  counter-clockwise 
traversal  of  the  contour.  Two  points  are  said  to  be  adjacent  if  they 
are  one  grid  cell  apart  in  a  horizontal,  vertical,  or  diagonal 
direction.  Figure  1  shows  the  eight  possible  neighbors  of  a  central 
reference  point  together  with  their  direction  codes  <2>. 

The  region  representation  will  be  obtained  by  a  sequence  of 
operations  performed  on  the  boundary  data  such  as 

(1)  gap  filling, 

(2)  local  smoothing, 

(3)  finding  global  extreme  points, 

(4)  fitting  linear  or  curvilinear  line  segments  between  extreme 
points , 

(5)  computing  the  centroid,  area  and  perimeter  of  the  region. 


l'\  ^  adjacent  to 
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Figure  1.  8  -  neighborhood. 


(6)  conputing  thf*  convex  hull,  and 

(7)  assigning  the  region  to  one  of  a  set  of  standard  shapes. 

Operations  (1)  and  (2)  deal  with  local  features  of  boundary  segnents. 
Gaps  can  be  filled  by  coiputing  the  bias  line  segaent  between  the  two 
endpoints  of  a  gap  and  digitizing  the  interaediate  points  on  the  line 
according  to  eight-connectedness.  An  operator  for  saoothing  local 
extrema  is  presented  in  the  next  section  of  this  report.  Operations 
(3)  and  (4)  seek  to  divide  the  closed  boundary  into  meaningful 
segments.  Just  as  intensity  discontinuities  are  iaportant  in  edge 
finding,  so  curvature  discontinuities  are  significant  in  object 
contours  <4>.  Fitting  of  curvilinear  lines  achieves  information 
conpression  as  well  as  facilitates  ma ni pulaticn  at  later  stages. 
Operation  (6)  is  useful  in  identifying  concavities  in  the  figure. 
These  would  be  due  to  occluding  objects.  If  the  fitted  lines  of  the 
contour  are  described  relative  to  the  centroid  and  the  principal  axes 
<1>,  then  the  representation  will  be  location  and  rotation  invariant. 
Size  invariance  can  be  achieved  by  noraalizing  contour  segments  with 
respect  to  area.  Invariant  properties  will  reduce  the  problem  of 
assigning  a  standard  shape  to  the  region.  Current  research  is  devoted 
to  (4)  ,  (6)  ,  and  ( 7)  . 

The  operations  described  above  are  not  only  useful  for  processing 
boundary  points  obtained  from  the  segmentor  but  also  for  data  acquired 
by  manual  segmentation.  We  have  developed  a  number  of  programs  to 
hand-segment  regions  interactively  from  a  displayed  digital  image. 
The  purpose  of  this  work  is  to  construct  object  descriptions  for  ’•he 
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knowledge  base.  Given  the  resolution  of  iaages  it  is  often  eztreaely 
difficult  to  aanually  extract  straight  continuous  boundary  segaents. 

Later  portions  of  this  report  present  an  algoritha  for  generating 
figures  or  regions  froa  their  boundary  coordinates. 
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2.7  LOCAL  SnOOTHING  OF  DIGITAL  CONTOORS 

Erica  H.  Rounds _ 

In  extracting  shape  features  froa  region  boundaries,  a  large 
amount  of  information  is  contained  in  the  global  discontinuities  of 
curvature,  i.e.,  at  the  extreme  points  where  a  significant  change  in 
direction  occurs.  It  is  frequently  desirable  to  perform  a  prior 
smoothing  operation  on  the  boundary  points  to  reduce  the  number  of 
small  local  extrema.  These  nay  be  due  to  noise  or  to  the  particular 
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algorithas  used  at  lower  levels. 

k  very  siaple  technique  for  saoothing  is  "Hysteresis  sioothing" 

described  by  Mason  and  Cleaens  <1>,  In  this  approach,  x  and  y 

coordinates  of  the  boundary  points  are  treated  separately.  The 

SBOothed  curve  essentially  follows  the  digitized  curve  within  a 

threshold  of  2k,  where  A  is  a  given  constant.  This  has  the  effect  of 

deleting  local  extrena  of  height  <  2A.  Let  Yj^^^be  the  y-coordinate  of 

the  {A+1)st  data  point,  and  let  y^^  be  its  snoothed  predecessor.  Then 

y  is  deterained  as  follows; 
k+1 


(i) 

If 

I'k+i  -  * 

'k+i  *  ‘ 

then 

'k+i*  ''k' 

<ii) 

If 

'k+i  '  ‘ 

>  ^k 

then 

'k  ■  *' 

(iii) 

If 

^k  > 

I'kti*  ‘ 

then 

'k+l'  'k  * 

Figures  la  and  1b  show  the  application  of  Hysteresis  saoothing  to  a 
curve  segaent.  The  dashed  line  indicates  the  snoothed  curve.  Note 
that  this  scheae  is  sensitive  to  the  spacing  cf  data  points.  In 
figure  1b  where  the  data  points  are  one  grid  cell  apart  (of  dimension 
A),  the  snoothed  curve  follows  the  original  cne  nore  faithfully. 
Thus,  hysteresis  smoothing  is  sensitive  to  gaps  in  digitized  curves. 
It  is  also  highly  dependent  on  the  starting  point.  Since  the 
smoothing  process  is  influenced  by  the  orientation  of  the  boundary 
segments,  it  will  distort  symmetry  inherent  in  the  figure. 
Furthermore,  for  closed  curves  the  smoothed  version  nay  not  be  closed. 
(For  an  example  see  figure  7a.) 

Figure  2  shows  a  different  type  of  smcothing  operation  called 


-45- 


here  '*P-S  pair  smoothln-j , ”  because  it  is  based  on  the  predecessor  and 


succassor  points  of  the  data  point  to  be  sioothed.  We  consider  all 
possible  P-S  pairs  in  a  3*3  neighborhood  of  the  reference  point 
(using  ei ght- neighborhood  adjacency).  There  ate  a  total  of  6U  such 
pairs,  of  which  the  eight  pairs  which  reverse  on  theiselves  have  been 
ignored.  That  is,  we  assume  that  the  boundary  encloses  a  region  at 
least  one  cell  wide.  The  remaining  56  pairs  have  been  classified  into 
six  types  as  shown  in  figure  3.  These  types  fora  a  basic  set  fros 
which  all  others  can  be  obtained  by  the  following  operations: 

o 

e j :  rotation  by  90  , 

^2^  rotation  by  **5°,  and 
e^:  reflection. 

Computation  of  types  is  very  efficient  since  for  each  type  (excepting 
I  and  V)  the  distance  a(P#S)  between  predecessor  P  and  successor  S  has 
a  unique  value.  Thus,  at  most  two  tests  are  necessary.  The 
properties  of  the  P-s  types  are  summarized  in  table  1  of  figure  4. 
Constant  A  is  the  dimension  of  the  grid  cell. 


The  objective  of  the  smoothing  operation  is  to  eliminate  local 
sharp  corners  such  as  types  II,  IVr  and  V.  For  II  and  IV,  the 
operation  deletes  the  central  reference  point.  In  case  of  type  V,  the 
reference  point  is  moved  to  create  a  type  I  P-S  pair.  Let 


Pi  =  (*.  »y.  )  denote  the  point  to  be  smoothed,  and  let 
be  the  smoothed  predecessor  of  Pi . 


TYPE  I 


TYPE  TL 


TYPE  11 


TYPE  12  TYPE  TYPE  ’31 


Figure  3.  Distinct  P-S  pairs. 
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2A 

21 

e,  eg  63  TOTAL 


Table  I 

Figure  4.  P,  S  Type  Characteristics 
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(0)  Initialize:  Scan  the  data  points  until  a  point  p.  is  found  sucb 
that  p.  has  type  1,  111,  or  ZV.  This  is  the  first  saoothed  point 
j  .  Set  i  to  i*1 . 

(1)  Coapute  a=  afPj_fPi+^  assign  a  type  to  p.  . 

(2)  If  type  =  I,  III,  or  IV  then  p^  “Pi  *  ^YP®  *  H  o* 

IV  then  set  i  to  i^-l  and  go  to  (1).  (This  checks  the  type  of  the  ne« 

reference  point  before  assigning  the  saoothed  point.)  If  type  =  V  thei 

coapute  p.  such  that  p.  has  type  I.  Go  to  (3). 

J  J 

(3)  Set  i  to  i4’1,  j  to  ji-l  and  continue  with  reaaining  data  points  at 
(1)  . 


The  perforaance  of  the  algoritha  is  deaonstrated  on  two  digitized 
contours  shown  in  figures  5a  and  5b.  Figures  6  and  7  coapare  the 
effects  of  Hysteresis  saoothing  with  P-S  pair  saoothiog.  In  general, 
P-S  pair  saoothing  tends  to  produce  a  curve  which  has  a  close 
reseablance  to  the  original  one.  Since  this  aethod  is  not 
direct  ion ‘dependent  it  will  preserve  syaaetry.  It  is  also  insensitive 
to  gaps  in  the  boundary,  since  successive  points  separated  by  aore 
than  2A  are  not  changed.  The  choice  of  starting  point  (aarked  ir 
figures  6  and  7  by  X)  assures  that  the  algoritha  will  produce  a  closed 
CO ntour . 

This  saoothing  algoritha  can  serve  as  a  preprocessor  before 
applying  aore  sophisticated  techniques  for  curve  fitting.  Althougl 
the  3x3  neighborhood  is  usually  not  large  enough  to  assess  th« 
global  significance  of  soae  corners,  it  is  felt  that  the  econoay  ol 
the  algoritha  outweighs  these  disadvantages.  The  large  nuaber  o) 
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Figure  6.  Smoothed  points  of  Figure  5.  a. 


possible  configurations  which  aust  be  considered  in  a  S  x  5  or  7  x 
neighborhood  would  add  considerable  coaputat icoa 1  coaplexity  withou 
producing  auch  better  results. 
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2.8  FIGURE  CONSTRUCTION  FROK  ITS  CONTOURS 

Erica  M.  Rounds _ 

A  aethod  is  presented  for  constructing  a  planar,  digitized  figui 
froB  its  closed  boundary.  This  can  be  useful  for  extracting  a  desire 
region  fron  a  given  scene  once  its  boundary  is  detecained.  Anothe 
potential  application  is  iaage  synthesis  where  the  iaage  i 
constructed  f roa  various  regions  to  which  different  intensity,  coloi 
or  texture  values  have  been  assigned.  Iaage  synthesis  cculd  be 
valuable  tool  for  coaparing  the  processed  picture  inforaation  with  ti 
original  input  scene. 

In  the  following  we  assuae  a  digitized  contour  usii 
eight-neighborhood  adjacency  and  counter-clockwise  traversal.  T 
algoritha  peraits  any  nuabec  of  "boles"  in  the  interior  of  the  figu 
or  region.  Hole  boundary  lists  are  siaply  appended  to  the  regi 
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boundary  list.  Figure  1  shows  the  boundary  points  for  a  region  with 
two  holes. 


The  algorithm  consists  of  two  steps.  During  the  first  step,  each 
boundary  point  assigned  a  type  which  is  stored  in  a 

corresponding  element  (i  ,j)  of  a  picture  matrix  P  having  dimension 

*  rr,4r,  *  ^  P  ®  baSCd 

max  min  max  min  k  k 

on  the  "shape"  of  the  line  segment  connecting  points  (x  -y  ), 

Jv  “  i  i\  “  * 

(:^,yj^),  ^*k+r'ic+l^*  Classification  into  types  is  straight-forward 

since  each  type  has  a  characteristic  pattern.  This  is  given  by  a 
vector  of  length  3  whose  elements  represent  the  number  of  neighbors 
above,  on  the  sane  horizontal  line,  and  below  a  given  reference  point 
in  a  3  X  3  neighborhood.  Figure  2  gives  the  pattern  and  typical  line 
segments  for  each  of  the  six  types.  Figure  3  shows  the  assignment  of 
boundary  types  to  the  points  in  figure  1.  Types  1  and  3  are  local 
peaks  (concave  or  convex)  ,  and  types  2  and  4  ate  end  segments  of  a 
horizontal  line.  The  latter  always  occur  in  pairs,  possibly  separated 
by  type  6  points. 


Step  2  is  described  using  figure  4.  The  interior  of  the  region 
is  indicated  by  hatching.  Each  row  of  P  is  scanned  left  to  right  and 
interior  cells  of  the  figure  are  filled  in,  i.e.,  corresponding 
elements  of  P  are  assigned  a  specified  value.  To  determine  the  left 
and  right  end  or  a  row  of  interior  points,  boundary  points  are 
examined  as  follows: 

(1)  If  the  boundary  point  is  approached  frem  the  exterior  of  the 
region,  then 
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Figure  3.  Assignment  of  boundary  types. 
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(a)  types  1  or  3  (figure  4f)  and  aatched  pairs  of  types  2 

(figure  4a)  are  convex  boundary  aegaents  which  are  ignored; 

(b)  type  5  (figure  4g)  and  unaatched  pairs  of  2  and  4  (figure  4c; 
the  beginning  of  an  interior  row. 

(2)  If  it  is  approached  froa  the  interior,  then 

(a)  types  1  or  3  (figure  4e)  and  aatched  pairs  of  types  2  i 

(figure  4b)  are  concave  boundary  segaents  which  signif 

continuation  of  an  interior  row; 

(b)  type  5  (figure  4h)  and  unaatched  pairs  of  2  and  4  (figure  4d; 
the  end  of  an  interior  row. 

These  cases  are  easily  distinguished  by  using  two  paraaeters,  oni 
the  interior/exterior  condition  and  one  for  reaeabering  w 
horizontal  boundary  segaent  is  scanned.  Figure  5  shows  the  resu. 
applying  step  2  to  the  boundary  types  in  figure  3. 

The  preceding  algoritha  requires  only  the  list  of  bo 

coordinates.  This  could  be  the  output  of  a  region  analyzer, 
coaputationally  quite  efficient  since  each  boundary  point  is  ex 
only  once  during  each  of  the  two  steps.  Storage  requireaen 
ainiaal  and  are  deterained  by  the  region  size.  The  resulting  p 

aatrix  can  be  aapped  into  any  desired  area  for  the  purpose  of 

synthesis. 

The  silhouette  generator  by  Dudani  <1 >  is  siailar  it 
respects.  It  operates  on  a  list  of  boundary  points  together 
list  of  predecesEor/suocessor  types  (for  a  discussion  of  thesi 
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Figure  5.  Generated  region. 
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section  2.6  of  this  report).  The  notion  of  a  •'region  crossing  poin 
includes  the  above  boundary  type  5  as  well  as  soie  other  cases.  T 
algoritha  requires  one  additional  step  and  cannot  be  used  directly 
regions  with  holes. 

Ref  ere  nee 

1.  S.h,  Dudani,  "Region  Extraction  Osing  Boundary  Following," 
appear  in  Proceedings  1976  IEEE  Workshop  on  Pattern  Recognition  a 
Artificial  Intelligence. 


[2.9  AERIAL  PICTURE  SYNTHESIS 
Benhaa  Ashjari _ 


laaqe  synthesis  involves  the  derivation  ky  coaputer  of  "bas 
inforaation"  froa  an  iaage  and  applying  it  to  recreate  the  pictu 
froa  a  set  of  available  pictorial  data  bases. 

The  research  currently  being  conducted  involves  digitized  aeri 
photographs.  In  soae  cases,  the  synthesized  version  of  a  picture  a 
the  picture  itself  are  close  enough  to  describe  various  applicatio 
of  iaage  synthesis.  In  general,  the  synthesized  picture  h 
statistical  and  visual  siailarities  to  the  original  one.  The  "bas 
inforaation"  derived  froa  the  original  iaage  suffices  to  describe  a 
fora  the  synthesized  version,  and  therefore  this  "basic  inforaation 
the  inforaation  which  represents  the  picture  of  interest  and  can 
stored  or  transaitted  instead  of  the  original  picture.  The  technig 
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involves  segmenting  an  area  into  distinct  sections  by  texture  or 
features,  and  preserving  only  the  outline  of  the  sections.  The 
displaced  segments  are  then  compared  with  a  predetermined  set  of  data 
bases,  and  those  most  similar  to  the  original  sections  are  chosen  for 
reproduction. 

It  can  be  shown  that  for  transmission  of  the  outline  of  a  picture 
and  synthesizing  it  at  the  receiving  station  we  need  0.1  to  0.2% 
(1000:1  or  500:1)  of  the  information  bits  that  would  be  required  to 
transmit  a  picture  in  the  usual  way. 


Aerial  Photographic  Data  Base 


The  data  base  used  here  is  a  set  of  128  x  128  aerial  pictures 
taken  from  a  flight  from  Los  Angeles  to  New  York  (see  figure  3). 
There  are  nine  distinct  scenes  in  the  data  base: 


1)  Airport 

2)  cloud  patterns 

3)  Desert 

4)  Farmland 

5)  Forest 

6)  nountain 

7)  Plains 

8)  High  density  urban  areas 

9)  Hater 


These  scenes  are  chosen  to  be  mutually  exclusive  (i.e.  such  that 
they  do  not  have  other  types  of  texture  or  scenes  in  then) .  For 
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Figure  1.  Original  Picture 
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Figure  3.  Aerial  data  base  used  for  synthesis  of  figure  1 
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exanple,  the  urban  area  does  not  include  a  rlTer  passing  through  i 
Figure  3  shovs  soae  of  the  128  x  128  data  base  pictures.  Ideally 
need  32  pictures  of  each  aerial  scene  (for  a  5  bit  code) ,  and  tt 
would  lake  the  total  nuaber  in  the  aerial  data  base  288.  However, 
have  a  total  of  91  now  available. 

Statistical  Analysis  on  the  Data  Base 

In  order  to  have  a  measure  of  closeness  between  the  data  ba 
pictures  and  the  corresponding  segmented  area  of  the  original  pictux 
we  need  to  set  some  natheBatical  criteria.  A  variety  of  aeasures  ba 
been  considered  for  coarseness,  dissimilarity,  homogeneity,  a 
orientation  which  all  fit  into  the  area  of  texture  analysis. 

Gray-Level  Co-occurrence '  ilatrix 

Before  analyzing  different  criteria,  we  need  to  discuss  1 
concept  of  gray-level  co-occurrence  matrix.  (In  the  literature  t1 
is  sometimes  referred  to  as  the  grey  tone  spatial  dependence  aati 
<3>. ) 

Consider  the  (x,y)  plane  and  a  picture  f(z,y)  defined  over  tl 
plane.  Let  A  =  (Ax, Ay)  be  a  vector  in  the  plane.  Be  can  compute  1 
joint  probability  density  of  any  pair  of  gray-levels  separated  by  i 
vector  A  .  For  the  discrete  case,  A*»Ay  ate  integers.  By  counting  1 
number  of  times  that  each  pair  of  gray  levels  occur  at  separation  i 
we  can  form  an  array  of  m  x  n,  where  m  is  the  number  of  gray  lev< 
present  in  the  picture.  In  this  case,  we  count  pairs  of  gray  lev< 
at  separation  A  or  -A  indifferently.  Therefore  a  symmetric  matrix 
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generated.  For  exaaple,  consider  a  5  z  5  pictore  as  the  following: 


Let 

-X) 

all 


112  2  3 

1  2  3  2  0 

0  10  0  3 

3  2  12  2 

2  3  2  1  0 

There  are  four  gray  levels  0«1,2,  and  3  in  the  above  picture. 
(dz,dy)  =  (1,0).  This  aeans  the  pairs  of  gray  levels  in  the  x  (or 
direction  and  with  a  distance  1  fron  each  other  are  counted.  For 
possibilities  the  gray  level  co-occurence  latrix  will  be: 

.0  I  ^  3 


2  3  11 

3  2  5  0 
15  4  6 
10  6  0 


We  generally  consider  the  vectors  to  be:  (1,0),  (1,1),  (0,1), 

(-1,1).  This  corresponds  to  the  pairs  of  pixels  with  0°,  45°,  90°  and 
o 

135  fron  the  x-axis.  Therefore,  there  are  four  (a  x  ■)  aatrices  for 
each  (n  x  n)  saaple  window  taken  out  of  the  data  base  (a  is  the  nuaber 
of  gray  levels,  n  is  the  nuaber  of  pixels  in  one  row  of  the  window). 
At  the  aoaent  n  is  chosen  to  he  32  and  we  scale  the  gray  levels  of  the 
pictures  to  be  32  levels  instead  of  the  noraal  256. 


.Texture 

Texture  analysis  plays  an  iaportant  role  in  iaage  synthesis.  To 
be  able  to  replace  a  part  of  the  original  picture  which  includes,  for 
exaaple,  faraland,  with  one  of  the  32  types  of  faraland  pictures  in 
the  data  base,  we  have  to  coapare  the  textures  in  these  two  scenes  and 
pick  the  closest  one. 


If  a  texture  is  coarse,  and  A  is  ssall  ccnpared  to  the  sixes 

the  texture  elenents,  the  pairs  of  points  at  separation  A  shou 

usually  have  sinilar  gray  levels.  This  neans  that  the  high  values 

the  natrix  should  be  concentrated  on  or  near  its  nain  diagona 

Conversely,  for  a  fine  texture,  if  A  is  conparable  to  the  textu 

elenent  size,  then  the  gray  levels  of  points  separated  by  A  shou 

often  be  quite  different,  so  that  the  values  in  should  be  spre 

out  relatively  unifornly.  Thus  a  good  way  to  analyze  textu 

coarseness  would  be  to  conpute,  for  various  values  of  the  nagnitude 

A,  sons  neasure  of  the  scatter  of  the  C  values  around  the  na 

A 

diagonal.  In  what  follows  P  (i,  J)  is  the  (i,  J)th  elenent  of  the  giv 
natrix  (which  has  size  a  x  n  s  32)  divided  by  a  nornalizing  factor 
which  is  equal  to  the  sun  of  all  of  the  natrix  entries  <4>.  P(i,J) 
actually  a  neasure  of  joint  probability  density  that  the  pairs  of  gr 
levels  (i,J)'s  occur  at  pairs  of  points  separated  ty  A=  (Ax, Ay). 


Measure  of  correlation: 

m 


m 


i=  I  j=  1 

fj  is  a  neasure  of  coarseness  <2> .  can  be  nocnalized  using  nea 

and  standard  deviations  associated  with  the  narginal  density  functio 

P^^”^^/P,  p^^’'^Vr. 

X  y 


Honogeneity  neasures: 

m  m 

E 

i=  1  J=  1 

(sun  of  squares  of  all  natrix  entries)  f^  is 
is  snallest  when  the  P(i,J)  are  all  as 


the  angular  second  none 
equal  as  possible.  It 


and 


large  when  soee  values  are  high  and  others  are  lov  <3,4>. 

m  m 

^3  =  53  53  log(P(i.  J)/R-) 

i  =  1  J=  1 

is  the  entropy  and  is  another  aeasure  of  hoiogeneity  in  a  picture. 
This  neasure  is  largest  for  egual  P(ifJ)  and  saall  whan  they  are  very 
unequal. 

Other  than  the  above,  the  radial  distributicn  of  Fourier  Power 
Spectrua  gives  a  aeasure  of  coarseness  but  was  ruled  out  because  it  is 
believed  to  have  a  high  coaputational  cost  and  poor  perforaance 
coapared  to  the  other  textural  features. 

Orientation 

Orientation  is  actually  another  aeasure  of  coarseness  and  it 
shows  that  a  texture  is  coarser  in  one  direction  than  another. 
Orientation  is  important  in  iaage  synthesis  because  of  the 
directionality  of  sene  natural  patterns,  e.g.,  aountains  or  aan-aade 
scenes  such  as  urban  areas  or  facalands. 

If  a  texture  is  directional  then  the  degree  of  spread  of  the 

values  about  the  aain  diagonal  in  should  vary  with  the  direction  of 

d  (assuaing  that  the  d  aagnitude  is  in  the  proper  range).  This 

texture  directionality  can  be  analyzed  by  coaparing  spread  aeasure  of 

C.  for  various  directions  of  d  <4>. 
a 

A  aeasure  for  directionality  has  been  considered  to  be  the 
contrast : 
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■atcix  around  its 


m  m 

^4  P(i,J)/R 

i=  1  J=  1 

This  is  essentially  the  aoaent  of  inertia  of  the 
■ain  diagonal;  it  is  a  eeasure  of  the  degree  of  spread  of  the  eatriz 
values.  It  is  expected  to  have  the  highest  value  at  the  direction 
perpendicular  to  the  directionality  of  the  texture. 

Also  the  "angular**  distribution  of  values  of  Fourier  Power 
Spectrue  is  sensitive  to  the  directionality  of  the  texture  in  f.  It 
will  have  its  high  values  concentrated  around  the  perpendicular 
direction  to  the  directionality  of  the  texture.  However,  at  this 
■oaent  it  is  not  being  considered  for  our  aeasureaents. 

Feature  Vectors 

A  32  X  32  window  is  taken  froa  each  data  base  picture  and  a  gray 

level  co-occurrence  aatrix  is  coaputed  for  each  of  the  four  directions 

0°,  45°,  90°  and  135°.  fj,  f^  and  f^  ace  coaputed  for  each 

aatrix.  There  are,  therefore,  16  nuabecs  associated  with  each  data 

base  picture.  These  16  nuabecs  can  be  assuaed  to  be  the  coordinates 

of  a  16-diaensional  vector.  Therefore,  there  are  32  vectors  foe  each 

class  (scene)  of  data  base  (D.,  ,D._,...D._J  and  we  have  nine  classes 

il  i2  i3Z 

in  the  data  base  (1  <  i  ^9).  These  nuabecs  will  be  saved  on  tape  foe 
future  use. 

Segaenta  tion 

The  first  stage  in  synthesizing  a  picture  is  delineating  th4 
distinct  areas  and  sectioning  then  according  to  their  texture  ant 
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features  (see  figures  1  and  2).  (It  sKould  be  aentioned  that  because 
of  unavailabilitf  of  a  simple  picture  which  could  include  various 
teitures  and  scenes,  the  original  data  (figure  1)  was  coaposed  such 
that  it  included  various  desirable  pictorial  data.) 

Segmentation  is  achieved  by  using  the  track-ball  on  the  digital 

* 

interactive  display  (Aerojet)  driven  by  the  PDP  1  The  locations 
of  the  pixels  over  which  the  track-ball  passes  is  determined  and 
stored  in  a  2000  x  2  array.  The  first  column  of  the  array  corresponds 
to  the  row-locations  and  the  second  column  of  the  array  gives  the 
column  location  of  the  pixels.  The  row  and  column  location  of  each 
pixel  is  put  in  the  array  as  the  track-ball  passes  over  then.  They 
are  the  pixels  which  are  located  on  the  contours  of  segmented 
pictures.  This  2000  x  2  array  along  with  the  description  of  what  are 
inside  the  contours  (mountain,  forest,  urban,  etc.)  are  sent  to  the 
user's  directory  in  a  POP  KI10  computer  on  the  ARPANET. 

The  user  can  now  start  synthesis  of  the  picture.  The  synthesis 
is  done  automatically.  The  user  should  answer  some  questions  asked  by 
the  program  on  his  terminal  and  the  operator  mounts  the  data  base  tape 
and  the  feature  vectors  tape. 

Statistical  Analysis  on  the  Original  Picture  and  Selection  of 
Appropriate  Data  Base 

*An  automatic  segmentation  routine  based  on  pattern  recognition 
techniques  is  being  developed  such  that  any  picture  nay  be  segmented 
by  computer  rather  than  manually. 
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Once  the  different  areas  have  been  isolated  (either  aanualljr  a 
autoaatica lly) ,  each  aust  be  aatcbed  against  a  wide  selection  c 
standard  data  bases.  For  example,  there  are  32  types  of  mountains  i 
the  data  base.  The  aountainous  area  in  the  original  picture  aust  t 
replaced  with  the  closest  of  these  32.  For  this  purpose,  we  have  t 
take  a  window  out  of  the  scene  to  be  synthesized  and  compute  the  gra 
level  co-occurrence  aatrix  in  four  directions  and  basically  do  th 
saae  type  of  calculations  as  described  earlier  and  cose  up  with  1 
numbers  which  constitute  a  feature  vector  Z.  Suppose  the  featur 
vector  is  froa  the  aountain  class.  He  have  to  compare  each  vecto 


D  ^  ,  D  ^  _  ,...,D  with  Z  and  pick  the  closest 

moxintainl  mountamZ  mountain  32 

No  scheme  has  been  developed  yet  for  this  purpose.  However,  th 
Fisher  linear  discriminant  technique  has  been  discussed  in  th 
literature  and  may  be  considered.  He  can  use  a  voting  scheme  t 
determine  the  closest  data  base  for  a  given  measurement  Z. 


/nthesis 


The  receiving  computer  has  the  following  information:  th 

segmentation  grid  and  the  data  base  numbers  with  their  respectii 
positions.  Nov  the  appropriate  data  base  is  selected  for  eac 
section.  If  the  computer  finds  the  data  base  smaller  than  the  spac 
it  is  to  fill,  it  automatically  magnifies  it  2x2,  using  a 
Interpolative  scheme  (see  figure  5) . 


The  computer  then  matches  the  grid  section  to  the  data  base 
cutting  off  marginal  areas,  and  pieces  all  of  the  segments  together 
Hhat  results  is  a  composite  picture  with  great  similarities  to  tl 
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original  (see  figures  4  and  5) 


Because  the  project  is  at  such  an  early  stage,  the  autcaatic 
selection  of  data  base  by  computer  was  not  possible.  The  data  bases 
in  figures  4  and  5  are  chosen  by  the  user  for  illustrative  purposes. 
The  choices  are  according  to  visual  siailarities  to  the  areas  in  the 
original  picture. 

The  problea  faced  is  that  in  a  synthesized  picture  roads  and 
rivers  may  appear  discontinuous  and  only  by  extensive  research  in  the 
future  can  we  hope  to  overcome  this  problea. 

Appl icat ion 

2 

For  any  n  x  n  picture  the  total  number  of  points  goes  as  n  , 
while  the  number  of  points  on  the  edges  is  proportional  to  n. 

For  transmission  of  a  256  x  256  picture  which  will  have  values 
from  0  to  255,  there  is  a  need  for  256  x  256  x  8  =  524288  bits  of 
information. 

If  this  picture  can  be  synthesized,  the  amount  of  information 
bits  needed  for  transmission  will  be  substantially  reduced.  This 
involves  segmenting  the  picture  to  its  differentiable  areas,  such  as 
mountains,  farmland,  etc.  This  task  is  achieved  by  drawing  a  contour 
around  each  area.  The  result  is  a  grid.  For  transmission  of  the 
location  information  of  the  contour  points,  on  the  grid,  a  great  deal 
of  advantage  arises  due  to  their  connectivity.  The  beginning  points 
of  a  contour  reguires  16  bits  to  specify  its  location.  Then,  through 
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Figure  4.  Synthesized  Picture 
of  Figure  1 


Figure  5.  Synthesized  with  2x2 
magnification 


The  above  figures  (4  and  5)  represent  the  simplest  form  of 
Image  Synthesis  for  figure  1. 

■’  i 


a  chain  coding  process  Me  can  specify  the  second  point  of  the  contc 
according  to  its  local  difference  with  the  first  one  and  £o  on. 
can  approxiaate  saall  variations  on  the  contour  vith  straight  lii 
and  therefore  we  have  to  code  only  the  iaportant  turning  points  on  1 
contour  and  with  this  procedure  we  probably  need  less  than  60  poii 
to  be  coded.  Considering  the  average  distance  of  two  points  to  be 
pixels,  we  need  8  bits  for  each  of  thea,  and  we  have  9  differc 
scenes  with  at  aost  32  variations  on  each  one  in  the  data  bases.  Ti 
takes  about  9  bits  of  inforaation  for  each  area  and  supposing  the 
are  1C  segmented  areas  on  a  picture;  for  transaitting  tl 
inforaation,  we  need  586  bits  which  is  between  0.1  to  0.2X  of  t 
inforaation  needed  to  transmit  the  entire  original  picture.  This  91 
to  99. 9f  saving  has  soae  applications.  For  exaaple,  in  soae  cas< 
the  local  relations  and  types  of  scenes  in  a  picture  are  of  concei 
not  the  actual  picture.  For  this  case,  iaage  synthesis  shall 
useful. 

In  soae  occasions  that  the  actual  data  is  needed  in  part  of 
picture,  synthesis  is  detriaental  for  that  part.  For  these  occasii 
that  part  aust  he  traosaitted  exactly  and  the  rest  can  be  syntbesiz 
For  exaaple,  in  figure  1,  the  airport  can  be  transaitted  exactly 
the  rest  of  the  picture  can  be  synthesized. 

In  ae teorological  applications,  original  clcud  patterns  can 
retained  while  the  exposed  surface  features  are  synthesized, 
synthesized  picture  will  be  an  exact  replica  and  there  will  si 
reaain  aany  cases  in  which  the  actual  photographs  ace  necessary. 
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these,  ieaqe  synthesis  vill  never  be  useful.  But  for  a  great  aany 
uses,  the  tiae  and  cost  saved  in  transiission  of  iaage  data 
coapensates  for  the  loss  of  originality  of  the  picture.  The  diagraa 
shows  various  stages  of  synthesis  and  tr ansa ission . 
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2.10  SOME  EDGE  SEGHEITATION  BESOLTS 


(RaaaAant  Hevatia _ J 

Satisfactory  segi<%otation  of  objects  in  a  scene  is  a  key  pact  of 
■ost  scene  analysis  systeas.  The  strategies  eiployed  for  segaeotation 
vary  along  many  diaensions,  often  deterained  by  the  particular  doaain 
of  the  pcoblea.  Here,  we  restrict  to  the  class  of  probleas  where  no 
prototype  specific  inforaation  about  the  scenes  is  available,  i.e. , 
the  objects  present  are  not  restricted  to  a  snail  set  of  objects,  nor 
any  infornation  known  about  their  spatial  relationships. 

Results  for  four  scenes,  shown  in  figure  1  are  presented  here. 
All  four  pictures  contain  only  a  few  (incidentally  only  one  each) 
objects  of  interest  and  the  objects  are  large  (occupy  a  significant 
proportion  of  the  picture) .  The  first  three  pictures  are  of  nilitary 
vehicles  against  a  desert  background.  The  chosen  pictures  represent 
perhaps  the  simplest  class  of  pictures,  useful  for  practical 
applications.  Yet,  by  the  current  state  of  the  art,  segmentation  of 
these  pictures  is  a  difficult  task  (at  least,  without  using  prior 
knowledge).  The  complicating  factors  are  the  presence  of  texture  and 
lack  of  multi-sensory  information,  such  as  cclct  or  range. 

Partial  segmentation  results  using  an  edge  based  approach  are 
presented  here.  Figures  2a  through  2d  show  the  edges  detected  in  the 
corresponding  pictures  by  the  application  of  a  Hueckel  Edge  Operator 
<1>.  The  operator  was  applied  at  every  secend  pixel  to  every  other 
row,  A  small  vector  along  the  direction  of  the  edge  is  displayed  for 
each  detected  edge.  Note  that  sane  thresholds  were  used  for  all  four 


-78- 


pictures  and  no  aanual  tuning  for  optiaua  edge  detection  wa 
attempted . 

The  edge  pictures  contain  a  large  nuaber  cf  edges  belonging  t 
the  textured  background  but  the  objects  ate  clearly  visible  to  th 
huaan  eye.  It  is  hypothesized  that  edges  belonging  to  object 
(particularly  aan-aade  objects)  tend  to  occur  along  elongate 
segaents,  whereas  aost  background  texture  edges  tend  to  be  distribute 
randoaly.  An  edge  linking  procedure  that  links  edges  with  siaila 
orientations  in  straight  line  segaents  along  chosen  directions  ha 
been  described  previously  <2>.  Results  of  applying  this  linkin 
procedure  to  edges  shown  in  figure  2  are  shown  in  figure  3.  Here 
only  those  edge  segaents  which  contain  at  least  seven  edge  eleaent 
are  retained. 

Of  course,  such  linked  edge  segaents  do  not  constitute  a  coaplet 
segaenation  of  the  scene.  However,  they  offer  proaising  startin 
points  for  further  segaentation.  The  next  step,  for  exaaple,  aay  bo 
coapletion  of  the  partial  boundaries  of  the  objects.  For  aan 
applications,  such  as  recognition,  the  partial  boundaries  nay  snffic 
in  aany  cases.  Research  is  in  progress  for  further  extending  thi 
approach . 

Results  shown  here  indicate  that  the  techniques  used  are  fair] 
successful  in  isolating  extended  edges  of  objects  of  interest.  Mai 
of  the  difficulties  arise  in  the  process  of  edge  detection  itsell 
HueckeL  edge  detector  is  unable  to  detect  fine,  high  resolution  edg< 
because  of  the  large  size  of  the  operator  and  also  fails  in  presen< 
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Figure  3. 


Linked  edge  segments  from  edges  of  Figure  2. 


of  fine  texture.  Local  high  resolution  edge  detectors,  and  al 
larger,  texture  edge  operators  say  be  used  to  suppleeent  the  ed^ 
detected  by  Hueckel  operator. 

A  aajor  aotivation  of  presenting  these  interaediate  results  is 
stiaulate  coaparison  of  techniques  used  by  various  groups  working 
the  ABPA  I  sage  Understanding  prograa.  It  is  felt  that  the  select 
scenes  are  good  test  vehicles  for  segaentation  techniques.  (A  larg 
nuaber  of  siailar  pictures  is  now  available  in  the  data  base  prepai 
by  DSC.) 

The  picture  used  in  figure  Id  is  a  sub- picture  of  the  house  see 
used  in  Ohlander's  work  at  CRO  <3>.  Ohlandcr's  technique  relies 
recursive  segaentation  based  on  histogra aaing  of  various  iaa 
attributes.  For  the  house  scene,  the  aost  effective  attributes  st 
to  be  those  based  on  color,  such  as  hue  and  saturation.  Mote  that  t 
results  presented  here  aake  no  use  of  color  inforaation  at  all.  (U 
of  color  in  segaentation  using  techniques  siailar  to  those  descrifa 
here  is  discussed  in  <4>. ) 
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2.11  EDGE  DETECTION  THROUGH  ORTHOGONAL  TRANSPOS HATICN 
Ikraa  E.  Abdou _ 

The  problea  of  edge  detection  was  discussed  through  nany 
different  approaches  <1>.  However,  so  far,  the  application  of 
orthogonal  transformation  in  edge  detection  has  not  been  given  enough 
consideration.  One  of  the  few  exaaples  in  this  trend  is  the  Huectel 
operator  <2>.  In  Hueckel’s  work  edge  detection  is  based  on  fitting 
the  intensities  in  a  given  region  to  an  ideal  edge  element.  The 
position  and  direction  of  this  edge  is  obtained  by  ainiaizing  the  aean 
square  error  between  the  intensities  of  the  region  considered  and 
those  of  the  ideal  edge.  Computation  is  simplified  by  transforaing 
into  the  Fourier  doaain  and  using  only  the  first  eight  coefficients. 
A  aeasure  of  the  goodness  of  edge  is  coaputed  and  is  used  in  accepting 
or  refusing  the  edge.  His  work  was  extended  tc  line  detection  in  <J>. 

Although  Hueckel's  results  are  quite  good,  the  operator  has  two 
disadvantages.  First,  it  is  time  consuaing  and  second,  snail  edges 


cannot  be  detected  in  the  relatively  large  block  size  used. 


Another  approach  to  the  problei  will  be  intioduced  here.  In  t 


approach  a  finite  set  of  edges  in  a  saall  block  is  considered, 
corresponding  Fourier  coefficients  are  conputcd.  Edge  detection 
iapleaented  by  coaparing  these  coefficients  to  suitable  thresholds. 

The  coaputation  of  Fourier  coefficients  is  a  siaple  problea, 
can  be  easily  generalized  to  shifted  or  rotated  edges.  This  is 
the  case  for  the  detection  problea,  where  a  statistical  aodel 
needed  to  obtain  an  optiaua  detection  strategy.  In  fact  the  w 
described  in  <4>  can  be  used  in  solving  this  problea. 


The  set  of  edges  considered  in  this  paper  is  shown  in  figure 

A  function  f  (x,  y)  defined  on  an  N  z  N  block  has  corresponding  Four 

coefficients  F(u,v)  given  by  , 

N- I  N- 1 

2  2 


where 


F(u,v)  =  ^  ^ 


(1) 


y=- 


N-l 

2 


x=  -■ 


N-1 


W  =  exp  (“ 


2tt  Y 
N 


(2) 


In  aany  cases  closed  foras  for  the  Fourier  coefficients  can 
derived.  As  an  ezaaple,  for  the  case  of  a  horizontal  edge  given  by 


£j^{x,y)  =  b 

Vi 

fory<  0 

(3a) 

=  b 

fory  =  0 

(3b) 

=  b 

+  h 

fory  >  0 

(3  c) 

The  corresponding  Fourier  coefficients  are 
- 

*N  is  dropped  for  siaplif ication. 


otherwise 


F^j(0,0)  =  N^(b  +  I  ) 


{«») 


Fj^(o,  v)  =  Nh 


(5) 


Fh(u»v)  =  0 


(if  u  0) 


(6) 


Fourier  coefficients  are  calculated  for  all  the  cases  shown 
figure  1.  The  results  for  N  =  5  are  shown  in  figure  2. 


Froa  these  results  it  is  obvious  that  the  edge  direction  can 
deterained  froa  the  Fourier  coefficients.  This  technique  is  used  wi 
the  test  iaage  given  in  <5>.  The  Fourier  coefficients  correspond! 
to  each  block  are  compared  to  different  thresholds  to  decide  t 
presence  or  absence  of  an  edge*  The  direction  of  the  edge, 
present,  is  also  determined.  Edge  naps  for  SHE  »  10  and  1  are  she 
in  figures  3  and  4,  respectively.  In  figures  3a  and  4a  only  t 
infornation  about  edge  location  is  used,  while  in  fignres  3b  and 
infornation  about  the  edge  direction  is  used  to  enhance  the  edge  na 
In  both  cases  the  resulting  edge  naps  show  sone  degree  of  inproveac 
over  those  obtained  by  the  Sobel  operator  <5>. 
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2.12  STATISTICAL  DESIGN  OF  EDGE  DETECTORS 


I  k  ra  m 


Ablou  and  William  K.  Pratt 


A  wid“  variety  of  edge  detection  methods  have  been  devised  <1-3>. 

f 

Only  recently  has  there  been  any  attempt  to  devise  a  figure  of  merit  [ 

r 

for  edge  detector  performance  <4>.  This  report  summarizes  an  ongoing  S 
effact  to  develop  a  statistical  model  for  edge  detection  ^hat  can  be 
used  in  the  design  and  a  priori  evaluation  of  edge  detectors  <‘^>. 


Edge  "'odol:  Figure  1  illustrates  the  basic  model  for  a  local 
ideal  edge!  defined  over  a  3  x  3  block  of  pixels.  These  ideal  edges 
are  assumed  to  be  subject  to  additive  whj.te  Gaussian  noise  with 
standard  leviationo  .  '^he  probability  density  at  each  3x3  point  in 
an  observation  array  is  therefore  a  Gaussian  density  of  the  form 


G(a,o)  =  [  Zno^r^  exp|~^^~^^-  | 


(1) 


where  y  r'^presents  the  observed  pixel  amplitude  and  a  is  its  moan 
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Figure  1.  Ideal  Basic  Edges 


value  as  specified  by  figure  1. 


Linear  Edge  Detection:  In  a  linear  edge  detector  the  observe 


pixel  array  F(j,k)  is  aultiplied  by  an  enhanceient  aask  n(j,k)  over  a( 


I  X  L  window  to  produce  an  edge  gradient  function 
L  L 


F(j.k)M(j.k) 


j=I  k=l 


Since  the  gradient  operation  is  linear,  the  probability  density  of  th4 


« 

gradient  P  (G)  =i'^(u,o)  will  also  be  Gaussian  with  aean  and  standar^ 


deviation  given  by 

3  3 


F(j,k)M(j,k) 

j=l 


L  j=l  k=l  J 

where  the  overbar  indicates  an  enseabie  average.  Edge  detection  it 


performed  by  thresholding  the  aagnitude  of  the  edge  gradient  functiot 


G  such  that  I G  I  _>  T  implies  the  presence  of  an  edge,  and  |g1  <  f 
implies  the  absence  of  an  edge.  The  probability  density  of  thf 


gradient  magnitude  is 


P(Ig1)  =  [Jr  (u,a)  +i'„(-u,  a)] 


for  G  >  0, 


The  detection  procedure  outlined  above  is  entirely  analogous  t^ 


signal  detection  in  radar  and  coaaunication  systems.  The  performance 
of  such  systems  is  commonly  measured  in  terms  cf  the  probability  o| 


detection  when  the  signal  is  present,  and  the  probability  of  fals^ 


detection  (false  alarm)  when  no  signal  is  present.  Applying  thii< 
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tcchaique  to  edge  detectioB*  the  pcobabilit|  of  false  detection  is 


tommi  to  be 


p(*Gl  1  u=0)  dx  =  2  erfc 


[-J 


•here  ecf c[ • ]  represents  the  coepleeentary  Gaussian  error  function. 
The  probability  of  detection  can  be  obtained  by  coeputing  the 
probability  of  eating  a  correct  decision  given  the  presence  of  the 
corresponding  edge.  If  all  edge  orientations  are  equi-probable,  the 
probability  of  detecting  an  edge  of  any  orientation  given  that  an  edge 


is  present  is  given  by 


p(lGt  1  u=u)  dx 


vhich  reduces  to 


Pj3  =  erfc 


--m 


equations  (5)  and  (6)  aay  be  employed  to  detereine  the  optieal 
threshold  as  a  function  of  noise  level  to  achieve  a  specified  false 
detection  probability  or  true  detection  probability. 


■onlinear  Edge  Detection:  In  a  nonlinear  edge  detector  the 
observed  pixel  array  P(j»h)  is  coebined  in  soee  nonlinear  Banner  over 
an  L  X  L  vindow  to  produce  an  edge  gradient.  inalysis  is  possible 
only  if  the  fora  of  the  nonlinearity  is  specified,  hs  an  example  of 
the  statistical  procedure  consideration  will  be  directed  to  the  Sobel 
operator.  Bith  the  Sobel  operator  the  edge  gradient  is 

G=[X^+Y^]^  (7) 

where 
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X  =  [F(1,3)  +  2F(2.  3)  +  F(3,  3)]  .  [F(l.l)  +  2F(2,  1)  +  F(3,  1)]  (8a) 

Y  =  [F(l,  1)  +  2F(1,  2)  +  F(l,  3)]  -  [F(3,  1)  +  2F(3,  2)  +  F(3,  3)]  (8b) 


The  pcobahility  densities  o£  X  and  Y  are  given  by  the  Gaussian 
densities 

p(X)  =  ^(Uj^.yT?  o  (9a) 


P(Y)  =I{u^./TTa 


(9b) : 


where  u  and  u  are  the  means  of  X  and  Y,  respectively,  for  a 
X  Y 


particular  edge  orientation.  The  probability  density  of  the  edge 
gradient  is  then  found  to  be 


P(G)  = 


12o 


exp 


(10) 


where  u  =  u^  ♦  uf,  and  I  (.)  is  the  modified  Eensel  function  of  zero 
AY  o 


order. 


With  this  statistical  model  the  probability  of  false  detection!: 

II 

^  I 

for  no  edge  present  is  i 


I- 


24o 


(11)  ii 


=  exp 

where  T  is  the  gradient  edge  threshold.  The  probability  of  detection 
becomes 

P..  = 


D  -  °|V?  I 


(12) 


where  Q{a,b)  is  Marcum's  Q  function  <6>. 


Conclusions:  The  statistical  design  procedure  summarized  in  this 
report  appears  to  hold  promise  for  the  design  of  edge  detectors  in  the 
presence  of  noise.  Further  study  is  underway  to  evaluate  the 
proced  ure . 
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ir.A(;E  PROCESSING  ERO.IECTS 


The  image  processing  projects  comprise  an  ongoing  research 
activi+-y  directed  to«ard  image  coding,  image  restoration,  vision' 
mololling,  and  the  implementation  of  imago  processing  systems.  In 
image  coiing  novel  ideas  based  upon  the  results  of  the  image 
understanding  study  are  being  explored  as  a  means  of  achieving 
significantly  higher  compression  ratios  than  obtainable  by' 
conventional  coding  methods.  The  image  restoration  studies  are 

directed  toward  the  solution  of  two  major  ptcblems;  blind  restoration! 

1 

in  which  a  priori  information  about  image  degradation  is  unavailable! 
or  incomplete;  and  constrained  restoration  which  involves  the  use  of 
luminance  hounds  and  smoothness  criteria  to  improved  image 

restoration.  Vision  modelling  research  activities  include  the 
extension  of  previously  developed  models  of  the  human  visual  system  to, 
encompass  higher  levels  of  visual  perception  in  support  of  the  image 
understanding  program.  Implementation  studies  are  underway  on 

tachnigues  of  nonlinear,  two  dimensional  optical  filtering  which  can 
he  utilized  as  a  form  of  sensor  based  image  processing. 


3.1  VARIABLE  KNOT  SPLINES  AS  AN  IHAGE  ANALYSIS  lECHNIQUR 
Dennis  C.  ncCaughey 

The  degrees  of  freedom  of  an  iaage  or  its  infcraation  content  is 
an  important  and  frequently  recurring  subject  in  iaage  processing.  It 
is  fundaaental  to  the  iaage  coding  problea  and  in  other  areas  such  as 
reaote  sensing.  (Here  reaote  sensing  refers  to  underwater  as  well  as 
PPV  and  satellite  iaaging.)  In  coding  problems  one  is  interested  in 
the  transmission  of  that  information  relevant  to  the  users  needs  and 
in  the  elimination  of  irrelevant  data,  while  in  the  remote  sensing 
problem  the  image  is  to  be  obtained  frcm  samples  of  a  corrupted 
version.  In  these  areas  the  concern  is  the  degrees  of  freedom  of 
sampled  images  and  in  this  context  there  have  been  several  attempts  at 
quantifying  this  notion  of  degrees  of  freedom. 

With  respect  to  the  remote  sensing  problem,  Twomey  <1>  has 
equated  the  degrees  of  freedom  of  samples  output  data  with  this  number 
of  non-zero  eigenvalues  of  the  gram-matrix  of  the  linear  system 
corrupting  the  original  functions.  This  idea  has  been  extended  in  two 
dimensions  and  applied  to  projection  imaging  systems  with  significant 
results  by  HcCaughey  and  Andrews  <2>.  However,  if  the  linear  system 
corrupting  the  original  function  is  unknown,  as  is  the  case  in  the 
blind  deconvolution  problem  <3>,  another  approach  is  necessary.  One 
such  approach  is  to  apply  the  singular  value  decomposition  (SVD) 
algorithm  <4>  to  the  sampled  image  matrix  whereupon  the  number  of 
degrees  of  freedom  can  be  equated  with  the  number  of  effectively 
non-zero  singular  values. 
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Both  of  these  aethods  deal  with  saapled  versions  of  the  iaage  on 
soae  linear  transf oroation  of  it,  and  are  thus  affected  by  the 
sampling  lethod  used.  In  the  SVD  case  this  is  readily  apparent  by 
considering  an  iaage  f(x,y)  that  can  be  written  as  the  product  of  two 
functions  fj(x)  and  ^  (*7)  =  Ijt*)  12^^^*  this  iaage  is 

saapled  on  a  Cartesian  grid  (x^,)^),  i  =  1,2, k  =  1,2, 

The  iaage  aatrix  can  be  written  as  the  outer  pioduct  of  the  two 
vectors  [f^  (x^)  ,..fj  (Xj^  ]  and  [  f^  (y^  . . .  f  2  (y  j>j)  ]  which  is  a  rank  one 
aatrix  for  all  such  separable  iaages  f.  The  point  is  that  the  degrees 
of  freedoa  should  be  a  characteristic  of  the  original  iaage  and| 
reflected  in  the  saapled  iaage  only  by  our  inability  to  collect  an! 
uncountably  infinite  nuaber  of  saaples  for  application  on  a  coaputer. 


This  brings  up  soae  conceptual  difficulties  since  the  nuaber  of 

degrees  of  freedoa  of  a  function  defined  on  a  continuua  is  countably 

infinite  at  best  viz  the  space  of  all  square  integrable  functions  on 
2  2 

[-1,1],  L  (-1,1),  Where  any  f  (x)  e  L  (-1,1)  can  be  written  as 

f(x)  =  Z  f;0:(x) 


i=  I 


1  1 


{0}  being  a  coaplete  orthonoraal  set  and 


f.  =  r  f(x)0*(x)dx. 

1  J  1 


Here  to  exactly  specify  f  in  an  L  sense  a  countably  infinite  set  {f.} 
is  required.  In  general  if  we  are  willing  to  accept  an  approxiaation 
with  soae  error  say,  d  ,  then  there  exists  a  nuaber  N(d,cp)  which  is  a 
function  of  d  and[q)},  such  that 


r  (.1  N(d^)  2  -le 

I  r  |f(x)  -  Z  rcpj(x)|  dx  <  d. 
^  -1  i=l 


We  could  then  define  the  degrees  of  freedom  of  that  f  at  level d  ,  DoF. 

9 

2 

as  the  inf  S(d,cp).  In  this  case  where  f  ei  (-1,1)  for  any  coaplete 
{cp} 
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set  {cp}  we  know  that 


so  that(S  is  a  monotonical ly  decreasing  function  of  the  nuaber  of 
basis  functions  used  in  the  expansion.  This  property  is  of 
fundamental  importance  since  without  it,  it  would  be  impossible  to 
compare  two  different  orthonormal  expansions  for  f.  In  what  follows 
we  will  generalize  this  concept  of  epsilon  degrees  of  freedom  using 
one  dimensional  cases  for  simplicity  and  then  extend  the  results  to 
two  dimensional  expansions. 


Let  f  be  an  element  of  normed  linear  vector  space  W  with  distance 

A 

function  d  .  Let  V  be  an  approximation  scheme  where  the  estimate  f  of 

fisgivenby  ^ 

f  =  E  J  . 

i=  1  ‘  " 

Here  N(<^,0,d)is  dependent  upon<S  ,  the  sequence  (  P)  and  the  metric  • 

We  further  require  the  property  that  vS  >  0  there  exists  such 

that  ri  ^  N  (  ,  ,d)  implies  that  d^(fX^)  <(S.  Further  we  requite  that 
be  a  monoton ica lly  increasing  function  of  the  precision  to 
which  we  are  approximating  f.  The  epsilon  degrees  of  freedom  with 


distance  DoF(d^<S),  can  then  be  defined  as  follows: 

DoF{d^,  (J)  =  infN((S,  p,  d^). 

To  find  the  set  tP]  is,  in  general,  far  too  much  to  hope  for.  However 


in  the  case  where  W  is  taken  to  be  the  space  of  bandlimited  L 


functions  observed  over  a  finite  subinterval  of  the  real  line,  the 

2 

distance  d^  being  the  L  me  tric,  the  result  is  known.  Here  for  every 
<S  the  ^n^  was  found  by  Landau  and  Pollack  <5>  to  be  achieved 

by  the  functions  {cp}  which  are  related  to  the  prolate  spheroidal 
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vaveforis  and  first  described  by  Slepian  and  Pollack  <6>.  While  this 

result  is  known,  the  de tereination  and  utilization  of  these  waveforas 

for  large  space  bandwidth  problews  has  aet  with  little  success. 

Therefore  to  arrive  at  any  aeaningful  and  useful  results  we  will  have 

to  restrict  the  search  to  functions  that  are  coaputa tlonally  feasible 

and  possess  the  appro xiaating  properties  required.  One  such  set  is 

the  set  of  polynoaial  splines  S  of  degree  k  with  n  free  knots  and 

k,  n 

defined  as  follows 

S,  =  {s(t)l  there  exists 

IVf  11 

0  =  *Q-  *2  “***~*r+l~  ^  integers  a2...a^ 

with  1  ^  a.  ^  k>1  for  each  i  and  a  s  n 

1 

such  that  s{t)  II  in  each  of  the  intervals 
...  k-m, 

ahxle  scc  i  in  an  open 
neighborhood  of  ,  i  =  0,...,r}. 


Here  II  denotes  the  class  of  all  polynoaials  of  degree  less  than  k. 
k 

S  thus  is  the  class  of  polynoaial  splines  allowing  aultiplicities  up 
k,  n 

to  order  k>1  at  each  knot  and  every  s  (t)  ®  ^  possess  the 

representation 


m. 
1 


s(t)  =  E  a.t^+  E  E  c..(t-x  ) 
i  =  0^  i  =  0j  =  0^-' 


n-j+1 


where 


n 


(x)!;  = 


X  a  0 


Clearly  the  method  of  fitting  a  different  pclyncaial  in  different 
subintervals  of  [0,1]  as  done  by  Pavlidis  <7>  can  be  classified  as  a 
pclynoaial  spline  with  appropriate  knot  aultiplicities  at  the 


subinterval  boundaries. 

Polynonial  splines  are  chosen  due  tc  their  approziaation 
properties  and  the  fact  that  they  possess  a  basis  naaely,  the  B-spline 
basis,  that  provides  a  local  basis  property  thus  allowing  a  rapid 
generation  while  the  matrices  involved  in  generating  a  B-spline  fit  to 
a  function  f  are  well  conditioned.  With  DeEoor's  algoritha  for 
computations  using  B-splines  <8>  no  difficulties  are  encountered  in 
handling  multiple  order  knots.  Hereafter  we  will  consider  a  spline 
s(t)  of  order  k  with  n  knots  s(t)  ®  S.  to  be  of  the  following  form 

tCf  71 

8(t)  =  E  b.Bj^^t) 
i=l  ^  '■ 

where  B'^'(t)  is  the  i-th  k  order  B-spline  basis  function, 
i 

p 

In  one  dimension  the  following  L  approximation  results  are  known 

<9>. 

Theorem  1:  Given  f  defined  on  [0,1]  then  there  exists  a  best  Lp 

approximation  of  f  of  the  form 

£  b 
i=  1  *  ^ 

Theoren  2  (DeBoor  unpublished):  Let  f  and  n  be  fixed  then  if 

_  -1  n  »  2 

si=  r  -  s  b.Bp^t)!  dt  ^  0 

k  ^  Jq  i=l  '  ' 

^  (k) 

where  X,  b.  B,  (t)  is  a  best  approximation  of  f  in  S.  then  • 

K.f  n  K+  i  K 

Thus  we  can  create  a  sequence  [s.  <t)}  of  best  approximating  splines  of 

order  k  such  that  if  s.  (t)  e  S,  ,  s.  „  (t)  e  S,  Then  s.  (t)‘*f(t)  in 

1  k,  n  i-n  k,  n+ 1  i 

2 

the  L  sense  as  i  goes  to  * .  Thus  polyncmial  splines  possess  the 
aforementioned  approximating  properties,  and  the  degree  of  freedom  at 
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level  epsilon  in  the  1  netric,  DoF  ,  11  •  ll^)^  will  be  taken  as 
PoF  (k,(f  ,11*1L)  =  Bintnls.  (t)£  S,  where  s.(t) 

b  i  ic  •  i 

2  2  , 
is  a  best  L  approx iaati on  to  f  e  L  [Orlli- 

The  de'jrees  of  freedoa  at  level  epsilon  can  be  siwilarly  defined 
in  two  dimensions  if  we  do  the  following.  Osing  the  B-spline  bases  of 
order  k  we  can  define  the  bilinear  spline  a pproxiwation  Sj^ 

'  ^  y 

f(xy)  as 

n  n 

f(xy)  =  E  £  b  „  (x,  y) 

i  =  1  j  =  1  ^  J  "x"y 

where  the  knots  have  been  defined  on  the  Cartesian  grid 
i  =  1,2,...,n  ,  k=l,2,...,n  .  The  degrees  of  freedom  can  then  be  taken 

X  y 

as: 

DoF  (k,<S  ,ll*|l2)  =  fflinCn^  ♦  "y^^n  ® 

2  2  *  y  , 

best  L  approxiaation  to  f  e  I  (0,1)]. 

For  our  preliainary  work  we  obtained  the  bicubic  spline  fits  froa  the 
following  equations 


n  n 

X 


1=  1  J  =  1  J  J 


k=  1,2, 

In  aatrix  notation  this  becoaes 

(3)i 


N  and  t  =  1,2, 


N 


(3) 

where  [B  ]  is  an  N  x  n  aatrix,  [b.,  ]  is  an  n  x  n  aatrix  and  [  B.  ] 

X  X  ij  X  y  j 

is  an  n  X  N  aatrix.  The  solution  ainiaizing  the  residual, 

^  -  2 

T  I  t  (».  .r.  )  -  £  (>|  .r.  )1  is  giT.n  bj 

For  each  experiaent  N  was  taken  to  be  128  so  that  the  effects  of  the 


ij 


quadrature  error  iaplicit  in  this  should  not  be  a  significant  factor. 
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1 


and  the  residual  should  be  a  good  appr oxiaat ion  to  the  L  error. 

For  our  prelioinary  work  we  investigated  the  possibility  of 
placing  the  knots  for  the  x  and  y  knot  sets  fro*  projection  of  f  (xy) 
along  the  y  and  x  data  sets,  respectively.  To  do  this  we  borrowed  an 
idea  suggested  by  DeBoor  <10>  wherein  he  suggests  placing  the  i*1-st 
knct  with  respect  to  the  i- th  knot  for  a  k-degree  spline  according  to 


the  following 


pVl  l/(k+l) 


dx  =  const 


where  f''(K)  indicates  the  i-th  derivative  of  f  (z)  .  To  deteraine  the 

x-knot  set  for  a  bicubic  spline  fit  the  knots  were  such  that 

*i+l  -41  i 

1  I  4J  y)dyj  dx 

and  sieilarly  for  the  y  knot  set.  The  results  for  a  bicubic  spline 
“50(x 

fit  to  f(xy)=e  with  10  knots  deterelned  in  this  eanner  for  each 

of  the  X  and  y  knot  sets  cospared  to  10  and  20  knots  taken  uniforsly 
are  shown  in  figures  1,  2,  3,  and  h.  The  corresponding  least  squares 
residuals  are  tabul::ted  in  table  1.  For  the  case  of  10  knots  the  knot 
placesents  are  shown  for  DeBoor 's  algor iths  and  for  unifors  knot 
placesent  in  figures  5  and  6,  respectively.  It  is  quite  evident  in 
figure  S  that  the  knots  have  been  concentrated  near  zero  where  the 
function  is  varying  aore  than  at  the  edges.  Froa  the  results  of  table 
1  it  is  clear  that  10  knots  placed  according  to  the  fourth  partials  of 
the  projections  is  clearly  better  than  the  unifora  case  for  both  10 
and  20  knots.  Figures  7,  8,  and  9  show  the  results  of  a  bicubic 
spline  approxiaation  to  an  actual  iaage  of  an  araored  personnel 
carrier,  for  40  unifora  knots  in  each  direction  in  figure  7  and  for  40 
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NUMBER  OF 
KNOTS 

PLACEMENT 

MODE 

RESIDUAL  SUM  OF 
SQUARES 

10 

UNIFORM 

2.  158 

10 

4th  DIFFERENCES 

1.29  X  10“^ 

20 

UNIFORM 

3.  24  X  10"^ 

— 

Table  1.  Residual  Sum  of  Squares  for  Knot  Placement  on 
Function  e“50{x  +y  ) 


/O?’ 


kncts  placed  according  to  the  fourth  difference  cf  the  projection  data 
in  figure  8.  Figure  9  is  the  original.  Table  2  contains  the 
respective  residuals.  The  respective  knot  placenents  along  with  the 
row  and  column  projections  are  shown  in  figures  10  and  11. 

Conclusions 


In  this  paper  the  attempt  has  been  to  define  the  concept  of  an  | 

"epsilon  degrees  of  freedom"  of  an  image  and  to  estimate  that  degrees 

of  freedom  by  the  number  of  variable  knot  bicubic  splines  necessary  to 

approximate  the  image  of  an  error  level  epsilon.  Considerable  success 

-50  2  2 

was  achieved  for  the  analytical  imaged  (x +y jwhere  10  knots,  each  in 
the  X  and  y  directions  were  placed  by  an  algorithm  dependent  on  the 
fourth  partials  of  the  projection,  and  provided  an  error  reduction  of 
three  orders  of  magnitude  over  the  situation  where  each  of  the  knot 
sets  were  uniformly  spaced  in  the  x  and  y  directions.  The  error  was 
such  that  10  knots  placed  in  the  above  manner  provided  an  error  lower 
than  that  achieved  by  placing  20  knots  uniformly  in  the  x  and  y 
directions.  For  an  actual  image  the  results  were  not  so  good  -  mainly 
a  result  of  the  difficulties  in  numerically  obtaining  the  fourth 
derivatives.  However,  while  the  residual  sum  of  sguares  is  greater 
for  the  variable  knot  case  than  the  uniform  koct  case  more  detail  is 
perceptable  in  the  variable  knot  bicubic  spline  approximation. 

Splines  were  chosen  as  the  expansion  functions  owing  to  their 
relatively  attractive  computation  and  properties  and  it  is  felt  that 
with  further  research  an  efficient  and  accurate  knot  placement 
algorithm  can  be  developed  which  would  provide  better  results  for  real 
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NUMBER  OF 
KNOTS 

PLACEMENT 

MODE 

RESIDUAL  SUM  OF 
SQUARES 

40 

UNIFORM 

1. 539  X  10^ 

40 

4th  DIFFERENCES 

1. 037  X  10^ 

Table  2.  Residual  Sum  of  Squares  for  Knot  Placement  on  APC 
Image 


//3 


inages  than  would  be 
approx in at ions. 


obtained  for 


unifocB  knot  bispline 
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3,2  A  TECHNIQOE  OF  A  PCSTEBIOEI  FESTORATXON 

John  Horton 


Introduction 


This  project  involves  the  restoration  of  a  blurred  iaage  without 
a  priori  knowledge  of  the  point  spread  function  (PSF)  of  the  degrading 
systea.  That  is,  consider ^the  aodel  of  the  degrading  systea 


g(x,y) 


h{h,  y,a. 


3) 


f(a,  8)dad8  +  n(x,  y) 


where  g(x,y)  is  the  degraded  iaaje,  h(x,y,a,8)  is  the 
f  (*»y)  is  the  iaage,  and  n(x,y)  is  noise. 


system  PSF, 
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The  aajority  of  cestocation  techniques  assuae  knowledge  of 


h(x,y,a>^)  and  estiaate  f(x,y).  Yet,  in  aost  cases  one  would  not  have 
knowledge  of  h(x,y,a.,3).  Thus,  the  technique  under  study  estiaates 
both  h(x,y,a,^)  and  f(x,y)  solely  fcoa  g(x,y)  while  assuaing  the 
following : 

1)  the  PSF  is  spatially  invariant,  i.e.,  h(x,y,a.,P)  =  hCx-a^y-P)^ 

2)  the  extent  of  the  PSF  is  snail  conpared  to  the  extent  of  the 
iaage,  and 

3)  the  iaage  is  not  so  severely  blurred  such  that  one  cannot  tell 
the  general  class,  for  exaaple,  building,  outdoor  scene,  etc.,  to 
which  the  blurred  iaage  belongs. 

Consider  the  Fourier  transfora  of  h 

F{h}  =  H(u,  v) 

Note  in  general  that  H  is  coaplex  and  can  be  represented  in  aagnitude: 
phase  fora  as 

H(u,v)  =  1  H(u,  v)l  e’ I 

Thus,  if  one  knows  |k(u,  v)|  and  6(u,v),  one  in  effect  knows  h(x,y)  via^ 
the  inverse  Fourier  transfora  relationship.  Previous  work  in  this 
area  was  teraed  "blind  deconvolution"  <l-4>  and  has  had  soae  aoderate 
successes.  The  work  of  Cole  <1>  estiaated  |  H(u,v)|.  Cannon  <2-4j 
extended  this  work  by  estiaating  |h(u,v)|  and  estiaating  3(u,v)  for  ^ 
set  of  three  coaaon  blurs.  The  estiaate  of  6(u,v)  was  accoaplished 
assuaing  the  blur  was  one  of  three  possible  blurs.  Using  a  patterij 
recognition  technique  Cannon  could  discriiinate  between  the  threaj 
possible  blurs  and  effectively  estiaate  Q(u,v).  The  present  work  doei 


not  assume  a  finite  set  of  possible  blurs  but  will  estimate  9(u,t) 
iirectly  from  <j(x,y).  The  mathematics  involved  in  the  estimate  of 
9(u,  V)  is  similar  to  the  mathematics  developed  by  Knox  [5-6]  for  recov¬ 
ering  astronomical  images  degraded  by  turbulance. 

Schematically,  the  a  posteriori  restoration  is  presented  in 
figure  1. 


Estimate  of  PSF  Magnitude 


The  estimate  of  |h(u,v)|  will  be  essentially  accomplished  by  the 
techniques  developed  by  Cole  <  1>  and  Cannon  <2>.  For  completeness, 
however,  let  us  review  the  methods  of  estimation.  For  reasons  of 
clarity  and  due  to  the  progress  of  the  work  to  date,  only  the 
noise- free  case  will  be  considered  herein.  By  assumption 


g{x,y)  =  //  h{x-a,  y-P)  f{x,y) 


=  h{x,y)  *f(x,y) 
where  *  denotes  convolution. 


If  the  blurred  image  g(x,y)  is  broken  into  smaller  images  g.  (x,y) 
such  that  the  spatial  extent  of  h(x,y)  is  less  than  the  spatial  extent 
of  each  g.  (x,y),  then  approximately 

gj(x,y)  «h{x,y)  *  fj(x,y) 

In  a  ddit  ion 

G,{u,  v)  «  H{u,  v)Fj{u,v)  (1) 

where  upper  case  letters  denote  Fourier  transforms  and  specifically 

1  GJu,v)l  «  1  H{u,v)!  1f^{u,v)|  . 
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Averaijing  r^er  the  ensemble  cf  subinages  where  <->  denotes 
averaging  over  quantities  indexed  by  i, 

<  1g^{u,v)1>  «  <  1  H(u,  v)|  1  F.(u,  v)l  > 

1h(u.v)1  <  I  F.(u,v)l  >. 

1 

For  classes  of  images,  for  example,  buildings,  it  was  found  <1> 
that  (u,v)l  >  was  very  similar  for  different  images  of  the  given 

class.  Thus,  an  estimate  of  |h(u,v)|  is 

,  <1g{u,v)1> 

1h(u,v)  ^  ^ - 

<  I  F.  (u,  v)l> 

where  <1(7  (u,v)|  >  is  performed  on  any  image  belonging  to  the  same 
class  or  averaged  over  members  of  the  same  class. 


Alternately,  if  one  assumes  stationar  it  y. 


where  i  ,  $ 
g  f 


i  (u,  v)  =  1h{u,v)1  *  (u,v) 

g 

are  the  power  spectra  of 


and  f,  respectively. 


Similarly,  it  was  found  that  f^(u,T)  is  very  similar  for  different 
images  of  a  given  class.  Thus,  a  second  estimate  of  H(u,t)  is 


1h{u,v)1 


♦  (u,v)  '  ^ 
g 

$^(u,  v) 


where  is  the  power  spectrum  of  an  image  belonging  to  the  same 
class. 


Estimation  of  PSF  Phase 

The  estimate  of  6(u,t)  proceeds  as  follows. 

jt,{u,v)  j9{u,v)  jq5.(u,v) 

Let  ua  conaider  the  autocorrelation 

* 

R^{u,  V,  Au,Av)  =<GJu,  v)  G.  (u+Au,  v+Av)>  (2) 
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Substituting  G.  (u,v)  fros  squatios  (1)  into  squatioD  (2) 

♦  ♦ 

R  (u,  V,  Au,  Av)  fv  <  H(u,  v)F  (u,  v)  H  (u+Au,  v+Av)  F.  (u+Au,  v+Av)> 

G  1  i 

=  H(u,v)H  (u+Au,  v+A  v)  <  F^(u,  v)F^  (u+Au,v+Av)>  (3) 

=H(u,  v)  H*  (u+Au.  v+Av)  Rj,(u,  v,  Au,  Av); [0 (u,  v)-0(u+Au,  v+Av)] 

=  1h(u,  v)|  I  H*{u+Au,  v+Av)|  e  Rj^(u,  v,  Au,  Av) 

From  equation  (1) 

^  I  R  (u,  V,  Au,  Av)  I  (4) 

|H(u,v)I  Ih  (u+Au,v+Av)|  ^  ' 

F 

CoBbining  equations  (3)  and  (4)  and  rearranging  results, 

:[0(u,  v)-0(u+Au,  v+Av)]  Rq(u,  v,Au,  Av)  |  Rp(u,  v,  Au,  Av)  | 

~  |Rq(u,  V,  Au,  Av)|  Rp(u,  V,  Au,  Av) 

and  it  follows  that 

-1^  R^  (u,  V,  Au,  Av)  |Rp(u,  V,  Au,Av)|  | 

0 (u+Au,  v+Av)  0 (u,  v).tan  J  j  |  R^(u,  v,  Au,  Av)  ) 

Assusing  one  knows  and  and  let+.ing  0(0,0)  =  0,  one  say 
obtain  0(u,t)  by  the  above  recursive  relationship.  Note,  however, 
that  one  can  calculate  R  fros  the  degraded  iaage  but  one  does  not 
know  R  a  priori.  Thus  to  estiaate  (u,v)  one  aust  estiaate  R  .  It 
has  been  assuaed  that  the  iaage  is  not  so  severely  degraded  that  one 
cannot  tell  the  general  class  to  which  it  belongs.  As  a  result,  the 
estiaate  of  R  will  be  calculated  by  using  a  siailar  clear  iaage. 
Thus,  the  success  of  the  phase  estiaation  technique  under  study  is 
contingent  on  the  siailarity  of  R  for  different  iaages  within  a  given 
class. 


Progress  to  Date 


A  series  ofc'  conputer  prograas  which  estiiate  both  aagnitade  and 
phase  of  the  PSF  has  been  written.  The  progcaas  are  being  run  on  test 


cases  to  ascertain  whether  or  not  the  prograas  are  error-free.  The 
programs  have  not  been  tried  on  real  images.  In  particular  the 

important  assumption  of  similarity  of  autocorrelation  R  for  different 
images  within  the  sane  class  has  not  as  yet  been  experiaentally 

verified.  This  ch<3ck  will  proceed  as  soon  as  reasonable  certainty  as 
to  the  correctness  of  the  computer  programs  has  been  determined. 


Although  the  tests  to  date  are  somewhat  artificial,  they  reveal 
problem  areas  and  illustrate  the  difficulty  of  obtaining  a  good  check 
of  the  computer  programs.  For  example,  consider  the  following  test 
case.  The  function 


f(x,  y)  =  ZOO  e 

was  calculated  and  stored  on  a  file  in  standard  eight  bit/pixel  image 
format.  This  simulated  image  was  then  convolved  with  a  function 
h  (X,  y)  and  the  result  stored  on  another  file  in  standard  eight 
bit/pixel  image  format.  Because  the  object  of  the  test  was  to  test 
the  computer  programs,  the  estimate  of  H(u,v)  was  then  calculated 
using  the  known  f  (x,y)  and  as  a  consequence  R  was  not  estimated  but 
was  calculat’d  from  the  known  f(x,y). 


Results  along  the  u  =  ^  and  v  =  0  axis  are  shown  in  figures  2  and 

3,  respectively.  The  correct  answers  are 

.  ,  8in(5TTu) 


rpspectively.  Note  in  both  figures  2  and  3  that  the  estimated  values  ; 
are  approximately  correct  and  in  general  the  error  increases  for  * 
increasing  u  and  v.  It  is  believed  that  the  error  is  due  to  the 
estimation  procedure  and  not  the  computer  programs.  Sources  of  error 
include  the  discretization  of  a  continuous  process,  limitations 
inherent  in  using  eight  hit/pixel  images,  and  inaccuracies  in  using 
the  discrete  Fourier  transform  to  approximate  the  continuous  Fourier 
transf  orn. 
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3.3  IBAGE  PROCESSING  BY  TRANSFORBS  OVER  A  FINITE  FIELD  USING  THE 
OVERLAP-ADD  SFC'»‘ICNING  TECHNIQUE 

Jin  Soh 

The  fast  Fourier  transfora  (FFT)  has  been  used  to  reduce  the 
coaputationa 1  cost  of  convolution.  Recent  work  has  deaonstrated  that 
there  are  attractive  analogies  to  the  FFT  in  certain  finite  fields  and 
rings.  Transforms  using  these  nuaber  theoretic  concepts  have  been 
developed  to  compute  convolutions  without  loss  of  inforaation  of 
sequences  usii g  the  FFT  algoritha.  The  transfocas  ace  defined  on 
finite  fields  and  rings  of  integers  with  the  arithaetic  carried  out 
aodulo  an  integer.  Because  of  these  characteristics  they  are  ideally 
suited  for  digital  coaputation  by  taking  into  account  quantization  of 
aaplitude  as  well  as  tine  in  their  definitions.  When  the  nodulus  is 
chosen  as  a  Fernat  nuaber,  transforaation  requires  only  on  the  order 
of  N  log  N  additions  and  word  shifts  but  no  aultiplications.  In 
addition  to  being  efficient,  they  have  no  roundoff  errors  and  do  not 
require  storage  of  basis  functions.  There  is  a  restriction  on  the 
length  of  sequences  inposed  by  the  word  length  and  also  there  is  a 
pcoblea  of  overflow  but  aethods  for  oveccoaing  these  are  being 


investigated.  Conparison  with  FPT  has  shovn  a  substantial  iaproveaent 
in  efficiency  and  accuracy.  Variations  on  the  basic  nuaber  theoretic 
transforas  are  also  being  developed. 

It  is  the  purpose  of  the  on-going  eiperiaent  to  point  out  that 
the  two-diaensional  nuaber  theoretic  transfora  aay  be  defined  as  a 
siaple  eitension  of  the  one-d iaensional  nuaber  theoretic  transfora. 
For  practical  cases,  two-diaensional  finite  iapulse  response  filters 
involve  iapulse  response  of  a  lot  fewer  than  ordinary  picture  size 
256  X  256  or  512  x  512  in  alaost  all  cases.  Since  these  nuaber 
theoretic  transforms  have  the  difficulty  that  the  word  length  required 
is  proportional  to  the  length  of  the  sequence  being  convolved,  it  is 
conceivable  that  sectioning  techniques  aay  be  of  value  when  sequences 
must  be  convolved  exactly,  such  as  when  ault iplication  is  very  costly, 
or  when  special-purpose  hardware  for  aodest  length  convolution  is 
being  considered.  For  the  large  picture  aentioned  above  the 
convolution  can  be  done  by  sectioning  the  picture  by  efficient 
cverlap-add  sectioning  deleting  and  end  effect.  For 
yin)  =  x(n)  *  h(n)  ,  we  assuae  that  x(n)  is  of  long  duration  or  that 
the  nuaber  of  saaples  representing  x  (n)  exceeds  the  size  of  the  h(n). 
Then,  since  the  shift  of  the  convolution  y(n)  is  siaply  the  sua  of  the 
shifts  of  the  functions  being  convolved,  no  inforaation  is  lost  if  we 
shift  each  sectioned  function  to  the  origin  prior  to  convolution.  is 
a  result,  it  is  necessary  to  decoapose  x(n)  into  sections  and  coapute 
the  discrete  convolution  as  aany  saaller  convclutions  to  avoid  the 
problea  of  dynaaic  range  of  the  nuaber  theoretic  transfora.  At  the 
saae  tiae,  if  N  is  saall,  then  we  would  have  the  advantage  that 
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considerable  tiae  saving  could  be  achieved  by  using  shorter  sequences. 
Therefore  it  can  be  true  that  the  use  of  nua ber  theoretic  transfora  as 
a  convolution  tool  aay  aake  it  possible  to  have  faster  and  aore 
efficient  picture  processing. 
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3.4  PHASE  CODING  FOR  OPTICAL  IHAGF  PROCESSING 
Chong- Kai  Useuh  and  Alexander  A.  Savcbuk 

flany  operations  in  digital  iaage  processing  such  as  restoration, 
linear  filtering  and  interpolation  can  be  perforaed  with  analog 
optical  systeas.  Research  in  these  fields  in  the  past  has  generally 
concentrated  on  coherent  optical  systeas  as  opposed  to  incoherent 
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With 


systeas.  With  the  use  of  general  aaplitude  and  phase  optica] 
eleaents,  aany  useful  iaage  processing  operations  can  be  done  vec] 
easily  vith  analog  incoherent  systeas.  If  general  optical  eleaenta 
coabining  aaplitude  and  phase  variation  can  be  aade,  the  only  aajol 
restriction  on  incoherent  systeas  is  that  the  iapulse  response  aust  be 
non- nega  tive , 


A  very  useful  function  suitable  for  an  incoherent  optical 
processing  systea  is  a  B-spline  interpolator  to  give  a  continuous 
desaapled  output  froa  the  discrete  spots  on  a  CRT  or  other  display 
systea.  Other  useful  operations  include  Wiener  filters,  high  spatial 
frequency  eaphasis  filters,  edge  or  directionally  sensitive  filters,! 
or  restoration  filters. 


The  filtering  operation  of  the  optical  systea  can  be  described  b| 
a  coaplex  aaplitude  point-spread  function  ii.(x)  or  non-negativ^ 
intensity  response  equal  to  the  aagnitude-squared  |ji(x)l^.  In  thJ 
frequency  or  pupil  doaain,  oaitting  scaling  factors  for  siaplicity^ 
the  desired  coaplex  pupil  function  li  (f )  is  given  by  ^ 

H(f)  =  ylh(x)}  (1)  j 


where  ]  denotes  the  Fourier  transfora  operation.  The  incoheren 
optical  transfer  function  *^(f)  is  then  described  by 


V(f)  =  H(f)  *H(f)=  y[|h(x)l  ]  ^2)  I 

using  transform  theory.  | 

With  t-hese  definitions,  the  probleas  and  tradeoffs  of  optical 
processing  can  be  defined.  Given  a  desired  h  (x)  or  |h(x)|^,  it  haf 


1 

I 
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been  difficult  to  directly  synthesize  anything  other  than  siaple 
functions  Jl(f).  Kith  the  developnent  of  sene  new  techniques  in 
coaplex  conputer  hologram  synthesis,  it  is  hoped  that  this  fundamental 
limitation  can  be  overcome.  One  major  approach  that  will  be 
investigated  is  to  use  the  precision  plotting  capabilities  of  a 
computer  controlled  microdensitometer  to  plot  amplitude/phase  masks 
directly.  Even  with  improved  capabilities  cf  these  areas,  the 
problems  of  spatial  frequency  response  and  dynamic  range  of  the 
hclogram  recording  medium  still  exist,  since  desirable  functions 
|t^(x)|  generally  have  transforms  which  reach  some  of  the  extremes. 
Here,  there  is  fortunately  a  free  variable  which  can  be  manipulated  to 
alleviate  some  of  these  problems.  This  variable  is  the  phase  <  h(x) 
of  the  impulse  response  which  is  entirely  free.  Thus,  the  major 
problem  of  incoherent  processing  is  to  chocse  a  pupil  function  H(f) 
which  has  low  dynamic  range  in  amplitude  and  phase,  has  low  spatial 
frequency  structure,  making  it  easy  to  plot,  and  gives  the  desired 
intensity  response  h(x)  accurately.  This  type  of  problem  is 
related  to  other  problems  in  image  processing  and  is  referred  to  under 
the  general  name  of  "phase  coding.” 

One  possible  approach  is  to  obtain  a  constant  amplitude  |H(f)| 
for  the  pupil  filter.  This  eliminates  the  need  for  any  amplitude 
variations.  In  digital  holography  the  resulting  pupil  filter  is 
called  a  kinoform.  One  approach  to  get  the  constant  amplitude 
spectrum  in  the  transform  domain  is  to  assume  a  phase  diffuser,  either 
random  or  deterministic,  for  the  phase  of  h  ( x)  .  Another  approach  is 
to  s»-art  with  a  random  phase  in  the  object  demain  and  go  through  a 
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recursive  searching  routine  to  reduce  the  dynaaic  range,  coaplexity. 


and  object  error.  Although  this  nuaerical  searching  is  a' 
tiae-consuaing  process  in  general,  it  does  show  better  results  than 
using  a  diffuser  alone.  In  the  application  of  analog  filtering,  the 
holograa  is  coaputed  only  once  and  the  coaputation  tine  is  no  longer  ai 
serious  problea  as  long  as  good  iapulse  response  has  bean  plotted. 

One  phase  coding  aethod  was  first  developed  at  IBM  <  1>  for  the 
coding  of  kinoforas.  Kinofora  errors  due  to  truncation  and 
quantization  are  discussed  by  Gallagher  and  Liu  <2>.  In  this  aethod, 
the  phases  of  object  are  assigned  randoaly  and  the  object  is  then | 

I 

Fourier  transforaed.  The  transfora  is  set  to  have  a  constant 
aaplitude  and  is  than  inversely  transforaed.  The  resultant  phase  is' 
used  as  the  new  phase  for  the  object.  This  procedure  is  repeated 
until  an  error  criterion  is  aet. 

Although  this  aethod  generally  gives  good  results,  it  has  been 
found  <3>  that  the  error  at  soae  points  fails  to  decrease,  stabilizing 
at  a  high  level.  This  prevents  the  error  froa  being  arbitrarily 
saall.  To  overcoae  this  deficiency,  Fienup  <3>  adopts  an  Input-Output 
Approach,  which  allows  the  aaplitude  of  the  input  as  well  as  the  phase 
to  be  Bodified.  This  gives  better  control  of  the  reconstruction.  In 
this  model  if  certain  output  is  used  as  input  then  the  sane  output  isi 
obtained.  Hence  modification  can  be  either  added  to  the  previous] 
input  or  present  output.  We  will  refer  to  these  two  nethods  as  Fienup 
I  and  Fienup  II,  respectively. 
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In  general,  the  shape  of  the  spectrua  does  not  have  to  be  flat. 
The  problei  then  becoaes:  given  the  aaplitudes  of  the  o’jject  and 
transfora,  how  do  we  find  the  corresponding  phases?  This  problea  has 
been  studied  by  Gerchberg  and  Saxton  <U>  and  the  aethod  is  essentially 
the  sane  as  the  iterative  method  aentioned  afcove  except  the  aaplitude 
of  the  transfora  is  set  to  be  the  desired  spectrum  rather  than  a 
constant.  Convergence  of  this  aethcd  is  also  proved.  This  aethod  has 
found  applications  in  aany  fields  such  as  aicroscopy  and  stellar 
interferometry. 

The  last  aethod  suggests  a  modification  to  the  hinofora 
procedure.  We  can  allow  the  aaplitude  of  the  transfora  to  vary  in 
such  a  way  that  the  amplitude  is  still  saooth  enough  to  plot.  Better 
results  and  faster  convergence  could  be  expected.  Since  the  spectrua 
can  always  be  expanded  in  a  Fourier  series,  the  sub  of  the  first  few 
large  components  can  be  chosen  as  the  approximation  of  the  spectrua. 
Here  we  can  consider  the  first  aethod  <1>  as  a  special  case  in  which 
only  D.C.  coaponents  are  chosen.  Note  that  Fienup's  method  cannot  be 
applied  in  this  general  approach  since  Fienup's  method  is  based  on  the 
assumption  that  the  amplitude  of  the  transform  is  constant.  Further 
derivation  is  require  i  in  applying  this  method. 

In  evaluating  the  results,  we  need  a  good  error  criterion  which 
can  bf  used  to  modify  the  computing  procedure  or  to  terminate  the 
procedure  without  human  intervention.  This  error  criterion  has  to  be 
consistent  with  the  human  visual  system.  In  general,  this  task  is  not 
possible  and  instead  the  mean  squared  amplitude  error  <2>  or  the 
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toot-BPan  squared  intensity  error  <4>  is  used.  Since  the  actual 
quantity  measured  is  the  intensity  of  the  image,  the  latter  one  light 
be  a  batter  criterion  although  it  is  larger  in  general.  We  can  also 
argue  that  since  huaan  eyes  are  sensitive  to  relative  intensity  other 
than  absolute  intensity,  it  would  be  better  tc  normalize  the  above 
quantities  so  that  these  quantities  are  not  affected  by  the  absolute 
intensity  and,  hopefully,  the  specific  object  used.  The  error 
criterion  adopted  in  this  study  is  the  following  for  an  N  x  II  picture 

r  1  ^  .  ,2  2  ,2  12  2  (3) 

^  2  /  ^  ^mn  "  ^mn  J  max 

.  '■N 

where  a  ,  T  are  the  resultant  image  and  the  desired  image, 

rrm  mn 

respectively  and 

T  =  Max  T  .  fm 

max  mn  '  ' 

m,  n 


One  dimensional  objects  are  used  in  the  experiment  for  simplicity. 
Only  one  of  them  will  be  illustrated  here.  Figure  1  shows  one  line  of 
a  picture.  This  is  a  2S6  point  array  imbedded  in  a  512  points  array 
in  order  to  smooth  the  transform  and  thus  the  kinofori.  Figures  2,  3, 
and  U  show  the  resultant  images  and  kinoforms  after  34  iterations 
the  methods  of  Hirsch  <1>,  Pienup  I  and  Fienup  II  <4>.  Compared  witH 
figure  1,  we  found  that  Fienup  II  is  slightly  tetter  but  almost  th^ 
same  as  the  first  method.  Fienup  I,  however,  gets  nearly  perfeci 
reconstruction  in  the  region  of  the  object  while  more  noise  ii 
introduced  elsewhere.  Since  intensity  is  measured  in  practice,  thii 
noise  in  amplitude  contributes  less  in  intensity. 


The  corresponding  errors  evaluated  by  eq.(1)  arc  given  by 
E,  =  .05222  E,  =  .01297  E  =.  04449 
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4 


This  data  is  consistent  with  the  results  shown  in  figures  2  to 
Modification  of  the  anplitude  of  the  transfom  has  also  been  tried.  A 
biased  cosine  function  is  chosen  because  it  is  siiilar  to  the 
spectrum.  The  result  is  in  figure  5  with  error  =  0.07162. 

Although  this  result  is  even  worse  than  the  first  method,  it  does  not 
eliminate  the  possibility  of  using  this  method.  Further  Fourier 
analysis  is  required  to  determine  the  component  to  be  picked  and  its 
corresponding  phase.  Perhaps  more  than  one  component  is  required  to 

get  better  results.  This  technique  remains  to  be  studied 

syst  emat ically . 

Experiments  so  tar  indicate  that  Fienup  IT  is  the  best  of  the 
methods  tried.  However,  the  performance  of  different  methods  changes 
from  one  object  to  another  and  since  other  methods  are  still  under 
test,  it  is  too  early  to  conclude  which  method  is  the  best  for 
particular  response  functions.  Future  work  will  include  tests  and 
analysis  of  the  various  techniques  and  application  to  two-dimensional 
problems . 
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3.5  ARTIFICIAL  STEREO  ALGORITHMS 

Alexander  A.  SawchuA 

In  a  previous  report,  the  technique  of  artificially  generating 
stereo  image  pairs  vas  introduced  <1>.  The  stereo  information  is 
obtained  from  various  image  features  or  combinations  of  features, 
including  pixel  brightness,  edge  information,  texture,  or 
multispectral  data.  The  technique  is  being  explored  as  an  enhancement 
aid  for  the  human  observer  whereby  these  features  are  mapped  into  the 
depth  perception  ability  of  the  observer.  It  should  be  emphasized 
that  the  stereo  pairs  generated  are  not  reality,  and  the  resulting 
stereo  pairs  nay  only  coincidentally  resemble  the  true  stereo. 

This  section  describes  some  of  the  details  of  the  algorithm  used 
to  obtain  the  stereo  and  outlines  current  applications  and  extensions 
under  exploration.  Figure  1  is  a  block  diajraiD  cf  the  entire  process. 
To  generate  the  artificial  stereo  pictures,  feature  extraction  such  as 
described  above  is  performed  on  a  single-frame  monocular  picture. 
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Linear  saoothimj  with  a  moving  average  convolutional  window  is  then 
performed  to  eliminate  any  noisy  or  jagged  edges  on  shifted  parts  of 
the  stereo  picture  pair.  The  averaging  window  is  necessary  to  avoid 
severe  shape  distortions  which  would  make  visual  registration  of  local 
features  difficult.  The  local  features  such  as  objects,  spots  and 
lines  must  remain  for  any  stereo  to  be  perceived.  Various  window 
sizes  have  been  tried,  and  it  appears  that  the  optimum  may  be  weakly 
pi ct ur e- dependent .  A  good  choice  for  most  situations,  however,  is  one 
with  d  pixel  horizontal  averaging  and  5  pixel  vertical  averaging.  The 
horizontal  averaging  is  kept  small  so  that  blurring  is  minimized,  and 
the  vertical  averaging  is  present  to  eliminate  the  most  objectionable 
vertical  jagged  lines  in  shifted  pixels. 

The  next  stage  of  the  operation  is  the  actual  stereo  pair 
generation.  Any  true  stereo  information  externally  supplied  can  be 
introduced  here,  and  a  gain  control  for  the  stereo  effect  is  provided 
in  the  algorithm.  The  stereo  effect  is  obtained  by  local  edge 
shifting  as  shown  in  figure  2.  The  perceived  stereo  in  the  left  eye 
is  obtained  by  locally  shifting  pixels  to  the  right.  For  the  right 
eye,  the  shift  is  in  the  opposite  direction.  Figure  3  shows  the 
details  of  the  shifting  procedure  for  each  line  of  the  image.  The  top 
graph  represents  the  desired  non-negative  height  information  after 
smoothing  as  a  function  of  position  along  the  image  line.  The  second 
graph  of  figure  3  represents  the  input-output  position 
characteristics.  The  height  data  is  added  to  the  unit  slope  linear 
function  shown  as  a  dotted  line.  With  no  height  information,  the 
dotted  line  would  be  a  napping  with  no  shift  from  input  to  output. 


-147- 


stereo  pair  generation 
left  eye  —  right  shift 
height  —  non-negative 
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Figure  3,  Shifting  algorithm 

The  necuni  q ra ph  is 


cur  v*»-  f  ol  loMi  fv 


then  Bade  monotonic  by  a 
alqorithm,  as  shown  in  the  bottom  graph  of  figure  3.  In  the  inpui 
regions  of  zero  slope,  the  pixels  are  obliterated  by  adjacent  onei 
moved  in.  In  regions  of  large  positive  slope,  certain  output  areal 
may  be  missed  and  left  blank.  When  this  happens,  pixels  are  moved  in, 
beginninj  from  left  to  right,  to  fill  in  the  blank  areas.  Thus,  I 
local  distortion  takes  place  to  create  the  local  shifting.  Thi 
directions  for  the  right  eye  are  simply  opposite. 

Future  work  on  artificial  stereo  will  concentrate  on  the  case  o: 
different  features  and  algorithm  improvement.  It  has  been  recent! 
found  that  the  addition  of  a  brightness  histogram  equalizing  step  ii 
the  block  diagram  of  figure  1  just  before  the  stereo  pair  generatioj 
gives  a  much  better  distribution  of  heights  and  a  more  pleasing  sterec 
illusion . 
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a.O  SMART  SENSOR  PROJECTS 


This  section  presents  the  results  c£  the  industrial  liaisonl 
between  OSCIPI  and  Hughes  Research  Laboratories.  The  objectives  ofi 
the  past  year's  subcontract  were  twofold:  first,  to  develop  real  tinej 
nonlinear  optical  data  processing  eleaents  and  second,  to  develop  a.: 
CCD  chip  for  saart  sensor  iaage  understanding  signal  processing.  Both; 
objectives  have  been  aet,  in  part,  with  guite  exciting  results 
proaised  in  the  latter  project.  All  projects  are  reported  herein. 
The  construction  of  the  CCD  chip  is  on  schedule  with  coapletion,  test, 
and  evaluation  expected  within  six  aonths. 


«l.  1  MOTIVATION  FOR  AN  ADAPTIVE  CCD  IMAGE  PROCESSING  CHIP 
Harry  C.  Andrews 

This  docuBent  represents  so«e  ideas  in  Botiwational  thought  which 
nay  be  of  some  use  in  expansion  of  the  Inage  Understanding  CCD  charge 
couple  device  signal  processing  chip  work.  The  chip  which  was 
suggested  for  implementation  under  the  first  year  subcontract  had  the 
Sobel  operator  amongst  four  other  different  operations  as  possible 
signal  processing  technigues  to  be  considered  for  implementation.  The 
motivation  for  these  five  different  signal  processing  techniques  may 
not  be  evident  to  the  CCD  implementors  and  therefore  this  report  is 
designed  to  describe  the  thought  processes  that  went  into  the 
generation  of  the  mathematical  formulation  for  the  function  suggested 
for  chip  implementation.  The  five  operations  so  suggested  are  listed 
below. 

1.  The  Sobel  operator 

2.  The  Mean  operator 

3.  The  Unsharp  Masking  operator  which  is  equal  to  the  Sobel 

operator  plus  a  weighting  of  the  Mean  operator 

iJ.  The  Thresholdod  Mean  or  Adaptive  Binarizing  operator 

S.  The  Adaptive  Stretching  operator 

These  five  operations  arc  described  mathematically  in  the  attached 
statement  of  work. 

The  Sobel  operator  is  essentially  a  convolutional  kernel  designed 
to  detect  edges.  The  operator  requires  a  3  x  3  convolving  matrix 
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which  is  designed  to  detect  wectical  edges  and  then  designed  to  detecf 


horizontal  edges,  the  two  outputs  of  the  wectical  and  horizontal 


detectors  then  being  coabined  by  an  absolute  walue  operation  to  fori 


the  Sobel  edge  output.  The  absolute  ralue  operation  is  in  lieu  of  a 


square  rooting  operation.  This  edge  detector  is  a  fairly  useful  one. 


requires  only  a  snail  kernel;  and  in  addition,  contains  nonlinear 


devices  (i.e.  the  absolute  value  operations).  The  reason  for 


selecting  this  edge  operator  is  because  of  the  nonlinearities  involved 


as  well  as  the  possibilities  for  future  inpleaentation  by  looking  at 


the  ratio  of  the  horizontal  to  the  vertical  edge  energy.  This  will 


provide  edge  direction  which  also  becones  a  useful  feature  technique 


to  be  utilized  in  higher  level  inage  understanding  operations.  The 


Sobel  operator  can  be  viewed  in  image  fora  and  should  provide  a  fairljl 


clean  edge  detected  iaage.  References  on  this  operation  are  readily! 


available  in  the  literature. 


The  nean  operator  is  essentially  nothing  aore  than  a  low  pasi 


filter.  h  3x3  matrix  was  suggested  for  i apleaentation  as  4 


convolving  kernel.  The  aean  operator  measures  the  average  brightness 


in  a  3  X  3  reg ion' surrounding  the  center  pixel.  This  operator,  as  i| 


stands  alone,  will  tend  to  blur  the  iaage  (remove  high  frequency  noisS 


as  well  as  removing  edge  detail).  The  operator  is  suggested  aS 


possibly  a  signal  processing  low  pass  filter  but  aore  significantly  aS 


a  switching  signal  to  allow  for  adaptive  signal  processing  iS 


subsequent  operations.  Because  the  nean  operator  tends  to  track  th^ 


low  frequency  variation  in  an  inage,  it  becomes  very  useful  fo( 
discriminating  between  regions  of  high  brightness  and  regions  of  larg^ 
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shadow  or  low  brightness. 

The  unsharp  washing  operator  is  suggested  as  one  technique  to 
utilize  with  the  wean  operator.  One  could  envision  a  weighting  of  a 
percentage  of  the  Sobal  output  with  a  percentage  of  the  wean  output, 
the  suw  of  the  two  percentages  equaling  100^.  Therefore,  one  could 
range  from  a  cowpletely  edge  image  to  a  completely  low  pass  or  blurred 
image  by  modifying  this  particular  percentage  weighting  factor.  If 
this  percentage  weighting  factor  were  in  fact  defined  or  determined  by 
some  information  content  within  the  image,  then  this  would  provide  an 
adaptive  signal  processing  aid.  However,  even  if  the  percentage 
weighting  factors  are  maintained  constant  or  fixed  throughout  the 
image,  this  technique  still  qualifies  for  that  under  the  general 
category  of  unsharp  mashing.  Unsharp  mashing  is  a  technique  utilized 
by  the  Associated  Press  in  transmitting  many  of  their  images  across 
telephone  communication  lines.  Unsharp  mashing  essentially  forms  a 
low  pass  version  of  an  image,  subtracts  it  from  the  original  image, 
weighted  by  a  certain  percentage  such  that  the  resulting  image  has 
higher  frequencies  emphasized  by  virtue  of  the  fact  that  the  low  pass 
or  blurred  portion  is  removed. 

The  fourth  signal  processing  function  suggested  for 
implementation  is  that  known  as  the  threshold  mean  or  the  adaptive 
binarizer.  This  function  has  numerous  applications  in  a  bandwidth 
compression  mode  of  operation.  Essentially  the  function  is  the 
following:  the  mean  operator  determines  the  local  area  mean  and  based 
on  the  local  area,  sets  the  center  pixel  or  pixel  in  the  center  of  the 
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identical  this  is  not  too  unfeasible  a  prospect. 

The  fifth  and  final  application  of  the  signal  processing  chip  is 
that  known  as  an  adaptive  stretching  operator.  Essentially  this 
operation  is  also  triggered  based  upon  a  decision  by  the  local  area 
nean  detector.  This  operation  is  hypothesized  by  virtue  of  the  fact 
that  in  regions  that  are  fairly  dark  one  would  like  to  stretch  the 
dark  or  low  aoplitude  brightness  values,  and  in  regions  that  are 
fairly  bright  one  would  like  to  stretch  the  large  or  high  aaplitude 
brightness  value.  Therefore,  this  particular  operation,  known  as 
adaptive  stretching,  is  one  in  which  the  lean  signal  is  tested  against 
a  constant  half  aaplitude  or  half  dynaaic  range  brightness  value.  If 
in  fact,  the  local  area  lean  is  greater  than  half  aaplitude  brightness 
on  the  average,  then  the  center  pixel  is  stretched  in  the  bright 
region.  If  in  fact,  the  local  area  aean  is  less  than  half  aaplitude 
of  the  dynaaic  range  possible,  then  the  center  pixel  is  stretched  in 
the  low  brightness  region.  This  operation  will  have  the  effect  of 
adaptively  selecting,  based  upon  bright  regions  or  dark  regions,  and 
stretching  the  appropriate  pixel  value  such  that  in  dark  regions,  low 
aaplitudes  are  enhanced  and  in  bright  regions,  high  aaplitudes  are 
enhanced . 

In  concluding  this  report  and  in  considering  the  signal 
processing  applications  for  the  chip  that  has  been  proposed  for 
developaent,  one  point  I  think  is  important  to  be  aade  clear.  Whether 
the  actual  signal  processes  that  have  been  described  becoie  useful  in 
an  iiage  understanding  or  higher  level  iaage  processing  environnent. 
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and  whether  the  processed  iaages  theBselves  are  pictorially  and  ; 
cosaetically  aore  pleasing  or  not,  is  really  not  the  point  of  the 

)! 

development  of  the  chip  under  consideration.  Rather,  the  selected 
five  functions  that  have  been  suggested,  themselves  represent  a  large  ^ 
variety  and  great  breadth  of  signal  processing  operations  that 

! 

conceivably  could  become  quite  useful  on  the  front  end  of  a  sensor 
device.  Therefore,  it  is  possible  that  all  the  implementations  will 
require  larger  kernel  arrays  or  modifications  of  the  signal  processing 
operation  described  thus  far.  However,  the  important  thing  to 
emphasize  is  the  fact  that  if  these  signal  processing  operations  can 
indeed  be  implemented,  and  with  the  linear  and  nonlinear  implications 
of  the  implementations  so  described,  such  implementation  is  a  success 
even  if  the  applications  results  are  not  striking  and  I  feel  that 
success  in  itself  will  motivate  further  investigation.  i 

Appendix  |i 

.  ^ 

) 

I.  There  shall  be  two  3x3  convolving  kernels  on  a  chip. 

kernel  i1  shall  be  the  Sobel  operator,  f  I 

kernel  2  shall  be  the  mean  operator,  f 

II.  There  shall  be  various  combinations  of  the  outputs  of  the  above  ^ 
kernels.  These  shall  form  five  different  output  lines,  and  therefore 
output  images. 

1.  Sobel  operator  =  edge  detection  =  f  (j,k) 

8 

2.  mean  operator  =  low  pass  filter  =  f^(j,k) 

3.  Sobel  ♦  a  mean  =  unsharp  masking  =  fu(j,k) 
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U.  Threshold  aean  =  adaptive  binarizer  = 

Function  leRocy  switch  =  adaptive  stretching  = 

ITT.  For  every  single  iaage  tested,  there  will  be  five  inages 
generated.  The  test  procedure  shall  be  either  in  real  ti«e  TV  or  via 
digital  eight  bits  512  x  512  lagnetic  tape  iiagery. 

natheaat ically  Defined  Functions 

1.  =  -I  {  1  f(j-l.k-l)  +  2  f(j,k-l)  +  f(j+l,k-l) 

-£(j-l,k+l)  -  2£(j,k+l)  -  £(j+l,k+l)  I 

+|£(j+l,k-l)  +2£{j+l,k)  +  £(j+l,k+l) 

.£(j-l,k-l)  -2£(j-l,k)  -  £(j-l,k+l)!} 

2.  £  (j,k)=4(  f(j-l,k-l)  +£(j-l,k)  +  f(i-l,k+l) 

m  7  * 

+£(j,k-l)  +£(j,k)  +  £(j,k+l) 

+£(j+l,k-l)  +£(j+l,k)  +  £(j+l,k+l)J 

3.  £^(j,k)  =  (1-a)  £^(j,k)  +  a  f^(j,k)  0  ^  a  ^  1 


4.  £^(j,k) 


r 

0 


fn,0,k)  s  £(j,k) 
£^0’»k)  >£(j,k) 
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2  Min  |f(j,k),-|  { 

2  Max  {f(j,k)--^  ),  0  { 


5.  f  (j,  k)  = 
a 


yj,k)=: 


2 


y(j,k)> 


2 


^(j*k)  =  center  pixel  in  3  x  3  array  to  be  processed 
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output  is  center  position  given  by  f^(j,k) 
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The  above  transfer  curves  are  to  be  switched  between  -  depending  on 
whether 


=  f  (j.k) 
m 


IS 


iOii] 

greater  or  less  than  r/2.  The  center  pixel  is  to  be  the  input 
^min(f(j,k),'|  )  ® 

kA 


r 


2max(f(j,k)-— ,  Oy- 


2min(f(j,k>,-|  )  |J 


2max(f(j,  k)--,  0)U 


1 


t|.2  CCD  IHIGE  PROCESSIHG  CIRCOITRY 

Scahas  Nudd 


During  the  past  year  ee  have  investigated  the  possibility  of 
perforeing  the  following  five  algorithes  with  CCD/HOS  integrated 
circuitry : 

1.  Chirp  transforsation 

2.  Robert's  cross 

3.  Sohel  operator 

4.  Hueckel  operator 

5.  Histogras  operation. 


We  have  developed  circuit  concepts  for  each  of  the  above  (apart  froe 
the  Hueckel  operator)  and  include  here  block  scheeatics  together  with 
a  brief  functional  description.  Details  of  the  circuit  design  and 
layout  are  in  general  onitted,  and  only  factors  which  directly  affect 
the  overall  concept  such  as  speed,  dynawic  range,  etc.  are  discussed. 
We  have  also  initiated  work  on  the  detailed  design  and  layout  of  two 


3  lines 
of 

image 


Same 
3  lines 
of. 

image 


test  circuits,  one  of  which  (the  Sobel  operator)  is  currently  bein^ 
fabricated.  Details  of  these  are  also  included. 


At  the  present  tiae  we  have  concentrated  our  efforts  on  chargfl 

transfer  technology  and  coapatible  HOS  circuitry,  because  of  ita 

-2 

inherently  low  power-delay  product  10  pJ  (versus  50  pJ  for  typical 
bipolar)  and  its  application  to  both  iaage  detection  and  processing. 
This  will  be  particularly  advantageous  for  the  developaent  cf  "Saart 
Sensors.”  A  concept  of  an  integrated  detector  and  pre-processor  ia 
shown  in  figure  1.  Here  a  CCD  iaaging  array  is  shown  as  detecting  tha 
iaage  and  creating  charge  carriers  which  are  then  clocked  through  a 
aatcix  of  CCD/nos  processing  circuits,  (which  light,  for  exaaple« 
pecfora  a  two  diaension  convolution  of  the  Sobel  type)  and  tha 
processed  data  stored  in  a  CCD  analog  store. 

Chirp  Transf oraation 

A  one  diaensional  CCD  iapleaented  chirp  transfora  has  bee! 
described  in  the  literature  <1>.  Factors  affecting  the  accuracy  an4 
speed  of  this  process  are  described  in  the  previous  seai-annual 
report.  The  aajor  issues  are  in  the  area  cf  (a)  integrating  tha 
peripheral  circuitry  such  as  clocking  circuits  and  input/outpul 
devices  on  the  chip,  (b)  providing  integrated  accurate  (^6  bitj 
aultipliers.  Both  these  areas  are  being  pursued  by  other  contractors^ 
and  hence  we  have  devoted  little  effort  in  this  area.  The  aajo( 
iapact  Hughes  Research  Laboratories  could  have  in  this  area  is  ta 
eaploy  its  high  resolution  electron  bean  exposure  techniques  ta 
fabricate  a  high  resolution  (S^B  bits)  CCD  aultiplying  digital  t( 
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analog  converter  <2>  as  previously  discussed.  Two  diaensional  FourieCj 

Transfora  perforaea  by  ccncatenatioD  of  tvo  one  diaension  transforas 

] 

requires  a  perauting  neaory  to  transfora  the  data.  Work  in  this  area 
is  currently  being  sponsored  by  the  Kaval  Undersea  Center. 

Robert’s  Cross  Operator 

A  circuit  which  will  perfora  the  Robert's  cross  operator  is  shown 
schematically  in  figure  2.  As  shown  it  consists  of  two  parallel  CCD 

I 

channels  into  which  adjacent  rows  of  picture  eleaents  are  fed.  The 

inputs  (not  shown  in  figure  2)  aight  typically  consist  of  the  Toapsett' 

I 

surface  potential  equilibration  circuits  <3>  or  the  CCD  circuit  aight 
also  be  used  as  the  sensor.  The  charge  corresponding  to  the  fouc^ 
adjacent  pixels  (a  through  d)  used  in  the  Robert's  cross  are 
si«ultaneously  sensed  by  floating  gate  electiodes  <4>.  These  outputs 
are  used  to  drive  a  nonlinear  CCD  circuit  which  perforas  the  aagnitude 
operation  1  a-d  1  ♦  lb-c|,  required  for  the  Robert's  cross  operation. 

A  scheaatic  of  the  circuit  is  shown  in  figure  3.  The  two  signals 
corresponding  to  pixels  "a"  and  "d"  are  connected  to  parallel  gates,] 
as  shown,  and  create  potential  wells  beneath  the  oxide  proportional  to^ 

I 

their  magnitude.  The  source  diffusions  are  then  pulsed  such  that' 
charge  flows  across  the  barrier  0^  and  into  both  wells.  For  surface 
channel  devices  the  surface  potential  0^  can  be  written  as 

i(  =  V  +  V  -  /v^  +  2VV^  (I) 

”s  O  VO  o  . 

where  v  =  -  Q/Co, 
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^♦;*x<oiaoc 


an  d 


V  a  ^  applied  gate  voltage 

O 

Q  =  charge  under  the  gate 
C  =  oxide  capacitance 

When  charge  flows  the  surface  potential  is  unifori  and  hence  the 
charge  stored  is  proportional  to  the  applied  voltage  (Q  =  .  For 

the  case  shown  in  figure  3  where  |d|  >  |a|  a  charge  proportional  to 
(d-a)  will  bo  stored  in  the  left  half  of  the  circuit.  Hhen  the  well 
shown  has  been  filled  excess  charge  will  flow  to  the  drain.  Should 
|al  >  |d|  a  charge  proportional  to  (a-d)  would  collect  in  the  right 
side  of  the  circuit.  In  either  case  after  the  source  voltage  is 
lowered  charge  proportional  to  |a-d|  is  duiped  into  the  central  bucket 
as  shown.  An  identical  circuit  is  used  to  form  |b-c|  and  the  full 
output  I  a-d 1  ♦  Ib-c  I  can  then  be  sensed.  The  output  circuit  is  shown 
as  a  simple  source  follower  and  reset  but  techniques  such  as 
correlated  louble  sampling  can  be  used,  if  necessary,  to  reduce  noise. 


Sobel  Operator 


The  Sobel  operator  operates  on  a  3  x  3  array  of  pixels  as  shown 
in  figure  U  and  forms  the  output 

fg(jk)  =  |lf(j  -  1.  k  +  1)  +  2f(j.k  -  1)  +  f(j  +  1.  k  -  1) 

-f(j  -  1,  k  +  1)  +  2f(j.k  +  1)  +  f(j  +  1.  k  +  1)| 

+  lf(j  +  1,  k  -  1)  +  2f(j  +  l.k)  +  f(j  +  1.  k  +  1) 

-  f(j  -  1,  k  -  1)  +  2f(j  -  l.k)  +  f(j  -  1.  k  +  l)l| 
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Two  basic  operations  are  involved  (a)  nondestractive  charge  tapping 
and  weighting  and  (b)  nagnitude  operation  and  suiiation  siiilar  to 
that  described  for  the  Robert's  cross: 

(a)  The  weighted  suaiation  can  be  perforaed  bj  varying  the  area 
of  the  sensing  gates  as  shown  in  figure  4.  Here  three  parallel 
analog  CCD  registers  are  shown  receiving  charge  frca  three 
adjacent  rows  of  the  picture,  and  the  gates  sense  an  output 
proportional  to  their  area.  For  clarity  cnly  the  storage  gates 
are  shown  and  the  clocJcing  lines  are  omitted. 

(b)  The  magnitude  operation  can  be  achieved  uy  a  number  of 
techniques  similar  to  that  described  for  the  Robert's  cross  and 
the  summation  by  charge  merging. 


Hueckel  Operation 

During  the  course  of  this 
investigating  the  HuecKel  operator 
analog  CCD  processing  cannot  direct 
algorithm  in  its  present  form, 
circuit  concepts  for  this  at  this  t 

Histogram  Operation 

A  block  schematic  of  the  circu 
real-time  histogram  are  shown  in 
data  is  used  to  set  up  a  potential 
CCD  and  a  linear  current  source  is 
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Figure  5.  Block  schematic  of  histogram  circuit. 


« 


m 


to  this  barrier.  When  the  charge  stored  is  sufficient  to  overcose 
current  flows  thereby  converting  aaplitude  to  tiae  delay.  The  charge 
crossing  the  triggers  a  pulse  shaping  network  which  feeds  data  into 
an  analog  store  thus  forning  the  histograa.  The  circuit  which 
provides  the  linear  current  source  is  shown  in  figure  6a.  Here  the 
first  stage  of  the  CCD  acts  as  a  virtual  drain  and  the  current  is 
self-regulating  via  the  feedback  path  around  the  aaplifier.  The 
charge  stored  after  tiae  t  is 

V 

Q(t)  =  -^  t  . 

For  an  input  signal  Vg  causing  a  barrier  charge  flows  when  Q(t) 

C  X  V  or  at  tiae  t  =  (V  /V  )  (C  R) .  If  the  shift  register  is 

os  s  o  o 

clocked  at  f  an  output  signal  appears  after  n  clock  cycles  where 


f  C  R 


This  data,  after  soae  pulse  shaping,  is  fed  into  a  serially  accessed 

digital  CCD  register  prior  to  parallel  readout  into  an  analog  store  as 

shown  in  figure  7.  The  accuracy  of  the  circuit  is  deterained 

priaarily  by  the  noise  in  the  input  circuit  and  error  accuaulating  in 

the  analog  store.  The  equivalent  noise  charge  in  the  input  circuit  is 

2 

/KTC  which  for  15  x  50  (ija)  gate  size  will  be  appro xiaately  150 

electrons.  For  0  bit  quantization  the  charge  storage  aust  therefore 

3 

be  in  excess  of  flO  x  10  electrons,  which  iaplies  a  gate  voltage  swing 

of  >1  V.  Since  the  current  from  the  input  circuit  shown  in  figure  6a 

•  0 

will  be  of  the  order  of  10  A  the  pixel  data  rate  is  Halted  to 

f  ^100  kHz.  requiring  the  aaplifier  to  settle  in  appioxiaately 
-8 

tt  X  10”  seconds.  Discrete  devices  are  currently  available  at  these 
rates  in  which  case  the  digital  CCD  register  would  be  required  to 
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(a)  LINEAR  CURRENT  SOURCE 


(b)  PULSE  SHAPING  CIRCUIT 
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(c)  REGENERATING  CIRCUIT 


Figure  6.  Circuit  elements  required  for  histogram  circuit. 


J73 


Figure  7.  Digital  CCD  register  and  analog  store 


operate  at  approxiaatelf  25  HHx,  which  is  about  the  present  state  of 
the  art.  Data  froa  this  register  is  shifted  out  in  parallel  to  the 
analog  store  (figure  7)  which  accepts  data  at  the  pixels  rate, 
100  kHz.  The  total  storage  tiae  is  practically  liaited  to  about  0.1 

seconds  by  dark  current.  Halting  the  operation  to  a  total  of  about 

4 

10  saaples. 

The  pulse-shaping  circuit  illustrated  in  figure  6b  and  6c  aust  be 

included  to  provide  a  logical  one  to  the  input  of  the  digital  shift 

register.  As  shown,  the  circuit  is  based  on  digital  refresh  described 

by  ICosnocky  and  Carnes  <5>.  After  charge  flews  over  the  barrier  a 

5 

string  of  logical  ones  with  charge  equivalent  to  approxiaately  10 
electrons  will  flow.  The  circuit  uses  floating  gate  sensors  to 

provide  a  single  pulse  output  coincident  with  the  start  of  this 
string.  (fhen  data  under  gates  D  and  E  are  both  ''zeros*'  the  refresh 
circuit  will  create  potential  wells  under  both  gates  A  and  B.  This 
will  cause  data  to  flow  to  drain  C.  Hhen  the  input  data  changes  (froa 
zero  to  one)  the  potential  well  under  gate  B  disappears  and  the  charge 
drawn  froa  the  source  is  clocked  through  to  F  prior  to  entering  the 
CCD  digital  delay.  When  both  D  and  E  are  "ones"  the  potential  under 
gate  A  rises  and  no  charge  flows  thus  providing  a  single  data  output 
at  a  tiae  equivalent  to  the  start  of  current  flow.  A  cross  section  of 
one  of  the  gate  interconnects  is  shown  in  figure  6c  together  with  the 
tiae  constants  involved.  Present  photolithographic  techniques 
indicate  that  this  circuit  will  operate  with  lelays  of  less  than  a  few 
nanoseconds. 
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Test  Circuits 


To  eaphasize  that  the  signal  processing  operations  described  here 
can  indeed  be  inpleaented  on  single  saall  chips  using  CCD/nos 
circuitry,  we  are  presently  supporting,  with  cur  own  internal  research 
funds,  the  developaent  of  test  circuits. 

Sobel  Operator 

As  shown  in  figure  8,  the  CCD  Sobel  operator  consists  of  a  two 
diaensional  CCD  filter,  CCD  absolute  value  circuits  and  a  charge 
suaaer.  The  input  format  is  three  parallel  or  demultiplexed  channels 
of  neighboring  scan  lines  on  the  image  plane.  The  sampled  analog 
charges  of  a  3x3  detector  submatrix  are  nondestructively 
transferred,  sensed,  added,  and  subtracted  in  a  mask-programmed  CCD. 
The  outputs  of  the  two  dimensional  CCD  filter  are  then  fed  into  the 
CCD  absolute  value  circuits  where  charges  corresponding  to  the  input 
magnitudes  are  gen<=rated.  The  final  output  is  the  charge  sum  of  the 
absolute  values  and  can  be  converted  into  voltage  forms  for  further 
processing. 

The  above  described  CCD  Sobel  operator  has  been  designed  on  a 
single  small  chip  which  might  easily  be  integrated  with  a  focal  plane 
detector  array,  thus  forming  an  integrated  sensor-processor  chip  which 
night  have  an  exciting  and  important  impact  on  future  image  pcocessing 
systems.  This  circuit  is  currently  being  fabricated  and  we  anticipate 
operating  devices  in  several  months. 

Test  Circuit  II 
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Five  basic  algorithas  are  addressed  in  the  second  test  circuit: 


1 .  Low  pass  filtering 

2.  Sobel  edge  detection 

3.  Unsharp  aasKing 

tt.  Adaptive  stretching 
5.  Binarizing 

A  block  schenatic  of  the  algorithi  is  shown  in  figure  9a  together  with 
the  five  separate  outputs.  It  is  designed  tc  operate  on  a  subarray  of 
3x3  adjacent  picture  elements,  and  a  simplified  circuit  concept  is 
illustrated  in  figure  9b.  A  brief  description  of  each  circuit 
component  is  given  below. 

low  Pass  Filtering 

the  low  pass  filtering  operation  is  achieved  by  the  parallel 
connection  of  nine  adjacent  floating  gates  as  illustrated  at  the  l<»ft 


of  figure  10. 

Here  ♦•hree  parallel 

CCD 

channels  are 

shown,  through 

which  thrae 

adjacent  lines 

of 

data 

a  re 

fed  (as 

for  the  Sobel 

operator) . 

For  simplicity 

the 

tra  nsfer 

gates  and 

clocking  and 

circuitry  are  omitted.  The  weighted  outputs  are  connected  to  a  single 
output  line  which  drives  a  source  follower  shown.  The  load  is  a 
variable  conductance  consisting  of  a  FET  with  variable  gate  voltage 
providing  an  output 

=  f  |f{j  -  1,  k  -  1)  +  f(j  -  l.k)  +  f(j  -  1.  k  +  1) 

+  f{j,k  -  1)  +  f(j,k)  +  f{j.  k  +  1) 

+  f{j  +  1.  k  -  1)  +  f(j  +  l.k)  +  f(j  +  1.  k  +  1)|  . 

The  tapping  technique  is  nondestructive  and  thus,  after  sensing,  the 


OUTPUT  (ii)  OUTPUT  (iii) 


Figure  9(a).  Block  schematic  for  test  Circuit  II 
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Figure  9(b).  Circuit  concept  for  test  Circuit  II 
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Figure  10.  Weighting  required  for  test  Circuit  II. 


charge  is  clocked  through  to  the  Sobel  operator. 


Sobel  Operator 

The  weighting  required  for  the  Sobel  operator  is  shown  on 

the  right  of  figure  10  (again  the  transfer  gates  and  clocking 
circuitry  are  omitted  for  clarity).  It  is  essentially  identical  to 
that  described  under  Test  circuit  I.  However,  the  input  charge  will 
be  clocked  directly  from  the  low  pass  filter  as  shown  and  the  output 
will  be  fed  to  the  charge  suiaing  circuitry  shown  in  figure  11. 


Unsharp  Haskinc 


The  unsharp  masking  output  f  (j,k)  =  f  (j,k)  ♦  af  (j,K)  is  formed 

us  rn 

by  the  charge  summing  circuit  shown  in  figure  11.  Here  charge 
corresponding  to  the  low  pass  filter  and  ^he  Sobel  operator  are 
clocked  under  a  single  storage  gate  as  shown  tc  form  f^(J»k).  Three 
stages  of  CCD  delay  are  shown  in  the  output  path  from  the  low  pass 
filter  to  equalize  the  charge  delay  prior  to  summation. 


•  tive  Stretchinq 


The  unsharp  masking  output  is  further  clocked  through  the  CCD  to 

a  barrier  )  which  is  controlled  by  an  externally  applied  voltage. 

If  the  signal  charge  less  the  the  charge  is  collected  on 

the  output  drain.  Charge  equivalent  to  Q  -  0.  /C  is  clocked  over 

usm  b  ox 

the  barrier  to  the  remaining  circuit.  The  effect  of  this  is  to  apply 
a  threshold  prior  to  a  variable  gain  output  connected  to  the  adaptive 
stretching  pad  shown. 
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Figure  11.  Illustration  of  layout  for  unsharp  masking  and  adaptive  stretching 


Adaptive  Binarizer 


The  binarized  circuit  illustrated  in  figure  12  is  based  on  the 
regenerative  charge  comparator  techniques  used  in  CCD  digital 
■enories.  It  consists  of  a  cross  coupled  flip  flop.  Charge  entering 
from  the  Onsharp  Hashing  operator  and  the  center  picture  elenent  are 
sensed  at  output  01  and  02,  respectively.  The  voltage  difference 
between  the  two  nodes  will  cause  differences  in  the  conductivities  of 
transistors  T1  and  T2  which  act  as  loads  for  the  cross  coupled  flip 
flop.  when  0,  is  pulsed  the  gate  voltages  on  T3  and  T4  therefore 
differ,  resulting  in  a  further  current  iabalance  which  rapidly  shuts 
one  transistor  off  while  driving  the  other  to  saturation.  The 
resulting  zero  or  one  is  taken  out  through  load  devices  (not  shown). 

Future  Plans 

Work  on  the  above  test  circuits  will  continue.  It  is  anticipated 
that  test  devices  for  Circuit  I  will  be  available  by  year  end,  when 
testing  will  coDinence.  We  have  devised  a  test  plan  which  will  use  an 
inSAI  8080  nic roprocessor  to  access  O.S.C.  data  tapes  and  also  drive 
the  CCD  circuits.  A  block  schematic  is  shown  in  figure  13.  After 
processing  the  supplied  data  will  be  displayed  as  shown  and  also 
supplied  on  nagnetic  tape  to  IJ.S.C. 

Because  of  the  wide  variety  of  linear  and  nonlinear  operations 
involved  in  Circuit  II  we  consider  that  a  demonstration  of  the 
feasibility  of  ^-his  circuit  is  most  important  and  we  have  large  impact 
in  the  community.  We  therefore  propose  that  this  circuit  be  developed 
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Figure  12.  Concept  of  binarizcr. 


Figure  13.  Schematic  of  test  setup. 


/?(> 


as  part  of  tho  next  phase  of  this  prograa. 
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4.3  REAL  TIME  NONLINEAR  OPTICAL  DATA  PROCESSING 

J.L.  Erickson,  A.  Au  and  J.  Grinberg 

During  the  past  year  ve  have  been  using  the  hybrid  field  effect 
liquid  crystal  light  valve  (LCLV)  as  an  incoherent-to-coherent 
interface  in  optical  data  processing  (COP)  experlaents. 


P 
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Siaultaneoasly ,  the  LCLV  has  functioned  as  the  thresholding/ saturating 
(clipper)  device  required  in  order  to  achieve  nonlinear  ODP  functions 
as  reported  by  S.R.  Dashiell  and  A. A.  Savchuk  (Optical 
CoBBun ica tions,  1^,  66-6^,  1*^75)  . 

The  structure  and  operation  of  the  hybrid  field  effect  LCLV  are 
described  in  detail  in  "A  New  Real-Tiae  Noncoherent  to  Coherent  Light 
Inage  Converter  -  the  Hybrid  Field  Effect  Liquid  Crystal  Light  Valve," 
J.  Grinberg,  et.  al..  Optical  Engineering,  14:3.  217-225,  Hay/June, 
1975.  This  device  was  developed  for  linear  OPD  applications,  and  so 
is  not  optinized  for  the  nonlinear  ODP  process. 

Our  approach  has  been  to  study  the  characteristics  of  the  LCLV  in 
a  level  slicing  experinent.  Level  slicing  is  a  difficult  nonlinear 
ODP  function  to  inplenent  because  it  requires  a  very  high  ganna 
response,  rapidly  saturating  clipper.  This  is  a  useful  approach, 
because  the  results  can  be  easily  Interpreted  in  terns  of  the  device 
character istics. 

The  ezperinental  setup  uses  the  coherent  optical  data  processing 
systen  shown  in  figure  1  with  variable  definitions.  The  projection 
light  is  derived  fron  a  nercury  arc  lanp  and  filtered  at  a  center 
wavelength  of  5250  angstrons.  The  coherent  light  source  is  a  50 
nV  He-Ne  laser.  The  densitonetry  curve  for  the  light  valve  operating 
at  4.0  volts  RRS  at  2  KHz  was  neasured  and  plotted  in  figure  2. 
HazinuB  ganna  was  2.1. 

Figure  3  is  the  transfer  function  achieved  in  a  level  slicing 


Figure  1.  Experimental  Setup 


Figure  2.  Densitometry  Curve  V  =  4.  0  volts  rms  at  2  KHz. 


experiment.  The  input  transparency  is  sampled  by  a  12  cycles/mm 
Ronchi  grating.  The  density  levels  of  the  grating  are  .21  ND  and  .68 
ND.  Although  the  light  valve's  gamma  is  too  low  to  achieve  an  ideal 
level  slice,  the  filtered  first  order  intensity  increases  '■o  a  peak 
where  the  input  intensity  at  high  gamma  has  maximum  modulation. 
Beyond  the  peak  the  first  order  output  intersity  decreases  as  the 
modulation  decreases  at  high  intensities.  Figure  4  shows  the  level 
slicing  function  with  the  light  valve  operating  at  5.8  vclts  PHS  at 
5  KHz. 


We  studied  the  LCLV  characteristics  that  impact  level  slicing 
performance.  '’’he  results  can  be  understood  with  reference  to  figure 
5,  which  shows  an  idealized  input/output  relationship  for  the  LCLV  ODP 
system.  The  LCLV  has  a  threshold  incoherent  input  intensity  I  below 
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level  slicing  experiments,  the  output  will  be 
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the  jratimj  mil  I  is  the  incilent  limp  intensity.  Repiaciny  the| 
inequalities  by  ejualitios,  the  qray  scale  (iensity)  ranqo  that 

a j  pears  as  a  single  "sliced’*  level  is 
-r,  ^max 

T  min 

The  total  iiuuiber  of  levels  that  can  he  i niepend ent  1  y  sliced  out 
defends  on  the  maxiiriim  lamp  intensity,  ^Lrnax*^^^  t  ra  nsi>c  rcnc  y  density 
range  covered  is 


D  ,  =  log 
total  ® 


T  I 
min  L  max 


1 


M 


and  the  number  of  levels,  n,  is 
^total 


■ 

These  equations  can  be  expressed  in  terms  of  quantities  that  are 

i 

characteristic  of  the  system  components  as  follows.  The  differential 
iensity  of  the  grating  is 


AD,,  =  log 


max 


min 


The  device  *Mensity'*  range  is 
AD  =  log  7-  . 

which  is,  in  a  photographic  analogy,  the  number  of  "log  exposure"! 
units  r^quirei  to  go  from  threshold  to  situration.  The  lamp  "df  nsity^j 
is 


log 


L  max 


i.e.  ,  the  number  of  saturation  "log  exposure"  units  available  from  thq 
lamp.  Finally,  the  tasic  grating  density  is 


D„  =  -  log  T  . 
G  min 


With  these  definitions,  we  have 


ADg  =  ADp  +  ADp 
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for  the  width  of  the  slice,  and 


^total 


for  the  total  density  ranye  that  can  be  sliced.  The  nuaber  of  level 


slices  in  this  range  is 


n  = 


^-■>0 


Therefore,  in  this  siople  picture,  the  nuaber  of  level  slices  can 
be  naxinized  by  using  a  low  density  grating  with  a  saall  density  step, 
a  bright  lamp  and  a  LCLV  with  a  saall  AOj^,  i.e.,  a  small 
some  extent,  the  systea  can  be  iaproved  by  increasing  the  device 
threshold,  T^,  if  the  intensity  range  I-j.  is  constant.  However, 

this  also  increases  therefore  decreases  tie  available  density 


range  D 


total 


The  threshold  intensity,  Irp,  is  deterained  primarily  by  the 
"switching  ratio"  of  the  LCLV  photocond uctor  (see  earlier  LCLV 
reference).  For  linear  operation,  a  low  threshold  is  desired,  and 
consequently  a  low  switching  ratio  is  used.  This  is  the  type  of 
levice  used  in  our  preliainary  experiments. 


A  higher  switching  ratio  is  obtained  with  a  thicker  CdS 

photoconductor  layer  in  the  LCLV.  Such  devices  are  now  being 
fabricated  and  an  improvement  from  ~  1  level  to  3-5  levels  is 

indicated.  Thus  far  we  have  not  been  able  to  obtain  a  thick  fila 

device  for  the  nonlinear  OOP  experiments,  though  attempts  are 

cont inuinq. 


Work  is  also  underway  (on  another  progtaa)  to  develop  silicon 


photoconductor  LCLVs.  The  silicon  characteristics  indicate  that  up  toj 


10  levels  could  be  isolated  with  the  saie  eguipaent  and  a  siliconj 

LCLV.  It  is  unlikely,  however,  that  a  silicon  device  will  be] 

■I 

available  for  tests.  i 

i 

The  most  lucrative  device  Dodif ication  is  to  find  liquid  crystal! 
■aterials  that  give  a  smaller  I,,*  I™.  This  would  reduce  AD,^and^ 
increase  D^,  both  of  which  would  increase  the  number  of  levels  n.  j 
There  is  a  good  probability  that  addition  of  some  cholesteric  liquid' 
crystal  to  the  device  mixture  would  provide  seme  improvement  in  j 

However,  the  required  liquid  crystal  chemistry  is  non-trivial  and - 
experiments  of  this  kind  are  beyond  the  scope  cf  the  current  program..] 

We  are  in  the  process  of  making  some  noise  measurements  on  the 
LCLV,  but  the  results  are  preliminary  and  will  not  be  reported  here.  | 


* 

t 
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5.  RECENT  PH.D  DISSERTATIOMS 

This  section  includes  the  abstracts  of  recent 

iissertations  completed  over  the  past  sir-month  research  period.  Dui 

i 

to  the  length  of  their  reports,  it  ms  felt  that  abstracts  presented 

I 

in  this  semi-annual  report  was  appropriate.  For  those  individual! 

1 

interested  in  further  details,  these  dissertations  appear  in  fheii 

I 

entirety  in  the  form  of  USC-Image  Processing  Institute  reports.  The] 
are  available  upon  request.  ^ 


I 


Coherent  optical  systens  are  of  interest  in  inage  or  data 
processing  because  of  their  ability  to  rapidly  handle  large  bandwidth 
data  in  parallel.  They  have  been  restricted  to  perforaing  linear 
operations  such  as  Fourier  transformation  and  convolution,  due  to  the 
inherent  linear  nature  of  an  optical  system  at  low  power  levels.  In 
♦■his  dissertation,  the  combination  of  a  ncnlinear  halftoning  step 
followed  by  band-pass  spatial  filtering  to  yield  a  specific  nonlinear 
intensity  transfer  function  is  explored. 

A  general  analysis  of  the  problem  assuming  infinite  copy  film 
gamma  and  saturation  density  is  made.  A  constructive  algorithm  for 
<lesigniag  a  halftone  cell  shape  and  selecting  the  diffraction  order  to 
yield  very  general  types  of  nonmonotonic  nonlinearities  is  presented. 
Mumerous  examples  of  the  synthesis  procedure  are  given. 

The  design  of  nonmonotonic  halftone  cells  which  allow  a 
nonmonotonic  nonlinearity  with  an  arbitrary  number  of  changes  of  sign 
in  slope  to  be  obtained  in  the  first  diffraction  order  is  considered. 
An  iterative  algorithm  suitable  for  computer  implementation,  and 
numerous  examples  of  halftone  cells  designed  with  this  algorithm  are 
g  i  ve  n . 

The  effects  of  allowing  the  film  gamma  and  saturation  density  to 
become  finite  are  analyzed,  and  a  technique  for  compensating  a  priori 
for  some  of  the  resulting  degradations  is  given. 
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Experiiental  results  with  general  halftone  screens  eade  on  a 
plotting  flatbed  aicrodensitoaeter  are  presented.  Logarithaic, 
exponential,  and  level  slice  characteristics  have  been  achieved  with 
aonotonic  cells.  Intensity  notch  filter  and  quantization 

characteristics  have  been  achieved  with  nonaonotonic  cells.  Other 

generalizations  of  the  technique  are  discussed,  including  the 

possibility  of  real-tiae  nonlinear  processing  with  optical  input  I 

I 

transd ucers. 

5,2  ESTIMATION  AND  DETECTION  OP  IMAGES  DEGRADED  BY  FILM-GRAIN  NOISE 

Pirouz  Naderi 

Fila-grain  noise  is  a  tera  describing  the  intrinsic  noise 
produced  by  a  photographic  saulsion  during  the  process  of  recording  an  , 

iaage  on  fils.  Although  fila-grain  noise  i  s  recently  been  considered  i 

within  the  field  of  iaage  processing,  the  nature  of  the  noise  is  still  I 
soaewhat  misunderstood.  I 

One  goal  of  this  study  has  been  to  use  the  theoretical  and^ 
experiaental  results  on  fila  characteristics  obtained  by  photographic' 
scientists  in  order  to  define  fila-grain  noise  within  the  context  of 
estiaation  theory.  A  detailed  aodel  for  the  overall  photographic 
iaaging  systea  is  presented.  There  are  linear  blurring  effects  at  the 
initial  and  the  final  segaents  of  this  aodel  to  account  for  various 
optical  and  cheaicai  degradations.  The  middle  segment  of  the  model 
represents  signal  dependence  effects  of  fila-grain  noise  and  includes 


Thp  accuracy  of  this 


a  nonlinear 
sinulatin<j 
of  siiilar 
coapar isons 
photogra  phic 


noise  tera.  The  accuracy  of  this  aodel  is  tested  by 
images  according  to  it  and  coaparing  the  results  to  iaages 
targets  that  were  actually  recorded  on  fila.  These 
point  out  that  the  aodel  is  a  gcod  representation  of  the 
imaging  systetc . 


’'he  restoration  of  images  degraded  by  fila-grain  noise  is 
considered  in  two  different  contexts  -  estimation  theory  and  detection 
theory.  Under  the  ♦•opic  of  estimation,  a  discrete  wiener  filter  is 
developed  which  explicitly  allows  for  the  signal-dependence  of  the 
noise.  The  filter  adaptively  alters  its  characteristics  based  on 
non-stationary  first  order  statistics  of  an  iaage.  This  filter  is 
shown  to  have  an  advantage  over  the  conventional  wiener  filter. 


In  the  cast)  of  extremely  low  contrast  images  digitized  by  a  very 
small  aperture,  filn-grain  noise  is  so  severe  that  conventional 
statistical  restoration  technigues  have  little  effect.  For  use  in 
this  situation,  a  heuristic  algorithm  is  developed  which  incorporates 
some  of  the  vision  properties  of  the  human  observer.  Bayesian 
detection  theory  is  used  to  justify  the  procedure  and  to  provide  some 
insight  into  its  use.  This  algorithm  also  explicitly  includes  the 
signal  dependence  on  the  noise  and  has  the  capability  of  greatly 
outperforming  the  human  observer  in  locating  objects  corrupted  by  very 
severe  noise. 


Fxporimental  results  for  modeling,  the  adaptive  estimation  filter 
and  the  Bayesian  detection  algorithm  are  presented. 
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5.3  IHAGE  RESTORATION  BY  SPLINE  FUNCTIONS 
Hohaaaad  J.  Peyrovian 

Spline  functions/  because  of  their  highly  desirable  interpolating 
and  approxieating  characteristics,  are  used  as  a  potential  alternative 
to  the  conventional  pulse  approxieation  aethod  in  digital  ieage 
processing.  For  uniforaly  spaced  knots,  a  class  of  spline  functions 
called  B-splines  has  the  useful  properties  of  shift  invariance, 
positi vaness,  convolutional  property  and  local  basis  property.  These 
properties  are  exploited  in  iaage  processing  for  linear  incoherent 
inaging  systeas. 

The  problen  of  iaage  degradation  in  a  linear  ieaging  systen  is 
described  by  a  superposition  integral.  For  sinulation  of  degradation 
and  restoration  by  neans  of  a  digital  conputer,  the  continuous  inaging 
■odel  Hust  be  discretizad.  Thus,  a  theoretical  and  experinental  study 
of  quadrature  fornulae,  particularly  nonospline  and  best  quadrature 
fornulac  in  the  sense  of  Sard,  is  presented.  It  is  shown  that  a  good 
choice  of  degree  for  a  nonospline  highly  depends  on  the  frequency 
content  of  the  integrand,  and  in  nost  cases,  a  cubic  nonospline 
generates  less  error  than  the  pulse  approxination  nethod  and 
Newton-Cotos  quadrature  fornulae. 

In  space-invariant  inaging  systeas,  the  object  and  point-spread 
function  are  represented  by  B-splines  of  degrees  n  and  n.  Exploiting 
the  convolutional  property,  the  deterainist ic  part  of  the  blurred 


—  . -  . T"  ft 


iaage  is  a  spline  function  of  degree  ■♦n+1.  h  niniaua  nora  principle 
leading  to  pseudo- inversion  is  used  for  the  restoration  of  space 
variant  degradations  and  un der deterained  and  overdeterained  aodels. 
Space- variant  point-spread  functions  that  describe  astigmatisa  and 
curvature  of  field  are  derived  and  coordinate  transforaations  are 
applied  to  reduce  the  dicensionality.  The 
singular- value-decoaposition  technique  is  used  for  solution  of  the 
siaplified  equations. 

For  noisy  blurred  images,  a  controllable  smoothing  criteria  based 
on  the  locally  variable  statistics  and  Biniaization  of  the  second 
derivative  is  defined,  and  the  corresponding  filter,  applicable  to 
both  space-variant  and  space- invariant  degradations,  is  obtained.  The 
paraaetcrs  of  the  filter  determine  the  local  smoothing  window  and 
overall  extent  of  smoothing,  and  thus  the  trade-off  between  resolution 
and  smoothing  is  controllable  in  a  spatially  nonstationary  manner. 
Since  the  matrices  of  this  filter  are  banded  circulant  or  Toeplitz, 
efficient  algorithms  are  used  for  matrix  manipulations. 


5.4  INTERFRAME  CODING  OF  DIGITAL  ir.AGES  USING  TBAHSFORK  AND  HYBRID 
TRANSFORM/PHEDICTIVE  TECHNIQUES 

John  A.  Roese 

In  the  design  of  digital  image  coding  systems,  th«*  principal 
objective  is  to  achieve  high  quality  receiver  ivage  reconstructions 
with  a  minimum  number  of  transmitted  code  bits.  Bit  rate  reductions 
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art*  achieved  by  exploiting  statistical  r3dundancies  within  an  iaage. 
This  is  coabined  by  transaission  of  only  those  portions  of  the 

i 

aa^.heoat  ical  iaage  representation  which  the  huaan  observer  is  aost 
sensitive  to.  This  dissertation  describes  research  intended  to  extend^ 
current  image  coding  techniques  to  the  coding  of  sequences  of  digital’ 
images  transaitted  over  a  digital  communications  channel.  The 
emphasis  is  directed  towards  definition  of  an  iaage  coding  system  that 
exploits  temporal  as  well  as  spatial  iaage  redundancies. 

A  primary  objective  of  this  investigation  is  to  develop  a  class 
of  interfraae  hybrid  transf ora/predictive  coders  having  near  optiaua 
levels  of  performance.  The  interfraae  hybrid  coder  implementations 
considered  employ  two-dimensional  unitary  transforms  in  the  spatial 
domain  coupled  with  first-order  DPCf»  predictive  coding  in  the  temporal' 
doaaiti.  Based  on  a  statistical  image  representation,  a  model  is 
developed  for  the  hybrid  coder  transform  coefficient  temporal 
difference  variance  matrix.  With  this  model,  theoretical  f!SB 
perforaance  levels  for  the  hybrid  coder  with  zonal  coding  ar« 
deterained  as  a  function  of  spatial  subblock  size. 

j 

I aplementa tions  of  the  interframe  hybrid  coder  using  discrete 
cosine  and  Fourier  transforms  are  experimentally  evaluated.  Hig( 
quality  image  reconstruction  are  demonstrated  for  reductions  of  32:' 
in  average  pixel  bit  rate.  Operational  considerations  investigate( 
for  the  hybrid  interfraae  coder  include  initial  conditions,  spatiai 
and  temporal  adaptation,  reinitialization,  and  total  transmission  bi 
rate.  Also,  sensitivity  of  the  interframe  hybrid  coder  to  channe 
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“rror  is 

Comparisons  arc  drawn  between  hybrid  transform/predictive  and 
three-d inonsiona 1  transform  interframe  coders.  Theoretical  zonal 
samplin'}  and  zonal  coding  HSE  performance  for  three-dimensional  cosine 
transform  coders  are  evaluated  for  different  frame  storage 
requirements  and  spatial  subbloch  sizes. 

A  tabular  summary  of  experimental  performance  and  system  design 
parameters  for  the  main  classes  of  interframe  coders  is  presented. 
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